DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Responsive to the communication dated 12/16/2025
Claims 1-6, 11-15, 19, 21-28 are presented for examination
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/16/2025 has been entered.
Response to Arguments - 101
Applicant's arguments filed 12/16/2025 have been fully considered but they are not persuasive.
Applicant argues that the claims are not directed to an abstract idea because they involve “the generation of image data through the combination of two images”
Examiner responds by explaining that a person could combine two images mentally by imagining both images and combining aspects of each into a single image, for example by drawing such an image with a pencil and paper. A person could, for example, imagine a dog and a human and draw a picture of a creature with the head of a dog and the body of a human. Such examples of imagery mashups have archeological evidence going back thousands of years. Similarly, a person could look at two images of fasteners, and draw a new third fastener using a pencil and paper that includes features of each of the observed originals.
The use of a generic CNN to do this, without any specifics as to how recited, amounts to no more than mere instructions to apply.
With respect to the remarks regarding ResearchCorp, the Examiner further clarifies on the prior statements -see MPEP 2106.04(a)(2)(III)(A) for “a claim to a method for rendering a halftone image of a digital image by comparing, pixel by pixel, the digital image against a blue noise mask, where the method required the manipulation of computer data structures (e.g., the pixels of a digital image and a two-dimensional array known as a mask) and the output of a modified computer data structure (a halftoned digital image), Research Corp. Techs., 627 F.3d at 868, 97 USPQ2d at 1280” and see MPEP 2106.05(a)(I)(iii) for iii. “A method of rendering a halftone digital image, Research Corp. Techs. v. Microsoft Corp., 627 F.3d 859, 868-69, 97 USPQ2d 1274, 1380 (Fed. Cir. 2010);” - i.e. it was that "the method required the manipulation of computer data structures (e.g., the pixels of a digital image and a two-dimensional array known as a mask) and the output of a modified computer data structure (a halftoned digital image)," that made those claims eligible. Such particular data structure manipulation is not present in the current claims.
In contrast, see Intellectual Ventures I v. Capital One Fin. Corp., 850 F.3d 1332, 121 USPQ2d 1940 (Fed. Cir. 2017) as discussed in 2106.05(f): "the steps in the claims described "the creation of a dynamic document based upon ‘management record types’ and ‘primary record types.’" 850 F.3d at 1339-40; 121 USPQ2d at 1945-46. The claims were found to be directed to the abstract idea of "collecting, displaying, and manipulating data." 850 F.3d at 1340; 121 USPQ2d at 1946. In addition to the abstract idea, the claims also recited the additional element of modifying the underlying XML document in response to modifications made in the dynamic document. 850 F.3d at 1342; 121 USPQ2d at 1947-48. Although the claims purported to modify the underlying XML document in response to modifications made in the dynamic document, nothing in the claims indicated what specific steps were undertaken other than merely using the abstract idea in the context of XML documents. The court thus held the claims ineligible, because the additional limitations provided only a result-oriented solution and lacked details as to how the computer performed the modifications, which was equivalent to the words "apply it". 850 F.3d at 1341-42; 121 USPQ2d at 1947-48 (citing Electric Power Group., 830 F.3d at 1356, 1356, USPQ2d at 1743-44 (cautioning against claims "so result focused, so functional, as to effectively cover any solution to an identified problem"
Similarly to Intellectual Ventures, there is very little specificity recited in the claims about how the images are generated besides generally the use of “executing a convolutional neural network”; as such the claims are also "so result focused, so functional, as to effectively cover any solution to an identified problem.”
Applicant argues that generation of an image using a neural network is not a mental process.
Examiner responds by explaining that while the operation neural network itself is not a mental process, the use of it to perform a mental process such as creating an image based on the features of other images at such a high level of generality amounts to no more than mere instructions to apply the judicial exception.
As for the generation of images, e.g. by drawing, or by using a computer as a tool, the Examiner notes that patent applications were submitted long before the invention of a computer with hand-drawn images, typically by a "draftsman", e.g. MPEP 608.02(b)(II) for FP 6.12: "New corrected drawings in compliance with 37 CFR 1.121(d) are required in this application because [1]. Applicant is advised to employ the services of a competent patent draftsperson outside the Office, as the U.S. Patent and Trademark Office does not prepare new drawings. The corrected drawings are required in reply to the Office action to avoid abandonment of the application. The requirement for corrected drawings will not be held in abeyance." And given that screws and the like predate a computer, one may have easily submitted a patent application for a new fastener, wherein one first submits drawings of the prior art fasteners, discussing certain details of the prior art, then combines the features of each (perhaps adding additional features) for an application on a new fastener along with the required drawings showing all parts of it. Further, the use of a paper and pencil to draw an image is fundamentally different from folding a paper airplane because it is not constructing a new physical structure; it is merely using the paper to store mental images imagined in the human mind. For example, a person could reasonably imagine a new type of screw that combines the properties of two other known screws; a person could choose a Phillips head bolt that is 20mm in length and a hex head bolt that is 5mm in length and, after observing both and making feature choices, create a mental depiction of a new fastener that has a hex head and is 20mm in length. These properties can be depicted in a multitude of ways, such as text describing them (i.e. “hex head; 20mm”), visually using an aide (i.e. drawing a hex head bolt that is 20mm long), or simply by imagining a mental image of such a combined fastener.
Applicants argue that the claims are eligible and provide an improvement because they are allegedly analogous to McRo.
Examiner responds by explaining that McRO was found eligible because it recited a process that automated a process that was previously only performable by hand due to the subjective nature of that process, importantly recited in a very particular way that was distinctly dissimilar from how the process was previously performed conventionally. Conversely, there is very little specificity recited in the present claims about how the images are generated besides generally the use of an RNN and CNN; how specifically the properties are generated and how specifically the images are generated using the respective neural networks is not described. Further, other than the use of generic neural networks recited at such a high level of generality that they can be considered generic components, it is not clear how this process is performed in a way that is fundamentally dissimilar from the mental process of observing the fasteners, identifying features of them, and drawing a new fastener combining features of both.
A mental process cannot be the basis of an improvement to technology as, being part of the abstract idea itself, it cannot integrate the abstract idea into a practical application nor provide significantly more. (MPEP 2106.05(a)(I): An inventive concept "cannot be furnished by the unpatentable law of nature (or natural phenomenon or abstract idea) itself." Genetic Techs. Ltd. v. Merial LLC, 818 F.3d 1369, 1376, 118 USPQ2d 1541, 1546 (Fed. Cir. 2016))
Response to Arguments - 103
Applicant's arguments filed 12/16/2025 have been fully considered but they are not persuasive.
Applicant argues that the change of “properties” to “property values” and the inclusion of a limitation reciting “wherein the set of estimated property values comprises one or more of a geometric property, a mechanical property, or a material composition property;” limits the property values in such a way that visual properties no longer read on them.
Examiner responds by explaining that, firstly, the use of the word “values” is not sufficient to limit such property values in such a way that visual values no longer apply; many examples of visual values exist (i.e. color, size, shape, etc.) Visual property values such as size and shape are particularly notable as examples of geometric properties. These properties are part of the image itself, i.e. during comparison if one image depicted a subject with vastly different size, shape, height, length, etc. than the other, the system would detect a lack of convergence. With this in mind, such visual comparisons as disclosed by the previously cited references, such as passages from Zheng ([Col 24 line 12-16] “At 1608, the methodology may calculate a visual similarity measure between the candidate product image portion and the input query image portion. At 1610, the methodology may output the visual similarity measure for use as a search result score for the candidate product.” [Col 19 line 56-58] “A visual search block 812 may calculate a visual similarity measure between input images, such as an image of a candidate product and the input query image.”) The comparison disclosed in Zheng is a comparison of the visual properties of images, i.e. a comparison of property values to other property values/ imagery. As mentioned above, if one image depicted a square and the other depicted a rectangle twice the width of the square in the first image, or depicted a triangle instead, this difference would be detected by the system of Zheng. Further, the use of the language “estimated property values” in particular enables a claim breadth that allows for the mapping of art that does not explicitly calculate exact numerical properties during this comparison, to which the examiner does not concede that Zheng fails to teach.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-6, 11-15, 19, 21-28 are rejected under 35 U.S.C. 101 because they are directed to an abstract idea without significantly more.
Claim 1 (Statutory Category – Process)
Yes, the claim recites a mental process, specifically:
MPEP 2106.04(a)(2)(Ill): “Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, Judgments, and opinions.”
Further, the MPEP recites “The courts do not distinguish between mental processes that are performed entirely in the human mind and mental processes that require a human to use a physical aid (e.g., pen and paper or a slide rule) to perform the claim limitation.”
A method comprising: obtaining an initial set of fastener parameter values for a fastener; executing a neural network model using features extracted at least from the initial set of fastener parameter values to query a fastener description repository; obtaining, from the fastener description repository, a set of possible matching fasteners; determining that a matching fastener does not exist;
Obtaining information in such a manner is a mental process equivalent to a person, such as a construction worker or carpenter, searching a hardware store for a certain type of fastener. The person may decide that, based on the needs of an ongoing project, they need Phillips head wood screws. Most hardware stores keep fasteners in organizers that group fasteners by type, such as intended use, head type, etc. that have visual keys to show where different types of hardware are stored. With this in mind such a worker would consult the key of the organizer, based on the desired fastener features, to find a collection of fasteners that meet their needs, such as a group of Phillips head wood screws of varying lengths. This kind of searching could also be done by observing a series of images of different fasteners, comparing the features of each, and judging if one or more meet the desired criteria. Determining that a matching fastener does not exist merely amounts to performing this search, having checked every option, and coming up empty handed. In other words, comparing each available fastener to the desired properties and judging, after exhausting all available fasteners, that none match these requirements.
See the July 2024 Patent Subject Matter Eligibility update: "A claim to “the collection of information from various sources (a Federal database, a State database, and a case worker) and understanding the meaning of that information (determining whether a person is receiving SSDI benefits and determining whether they are eligible for benefits under the law),” where “ `[t]hese steps can be performed by a human, using “observation, evaluation, judgment, [and] opinion,” because they involve making determinations and identifications, which are mental tasks humans routinely do,' ” and thus can practically be performed in the human mind, In re Killian, 45 F.4th 1373, 1379 (Fed. Cir. 2022). "
Also see in the July 2024 Patent Subject Matter Eligibility update: "Claims to “the use of an algorithm-generated content-based identifier to perform the claimed data-management functions,” which include limitations to “controlling access to data items,” “retrieving and delivering copies of data items,” and “marking copies of data items for deletion,” where the claims cover “a medley of mental processes that, taken together, amount only to a multistep mental process,” such that the steps can be practically performed in the human mind, PersonalWeb Techs. LLC v. Google LLC, 8 F.4th 1310, 1316-18 (Fed. Cir. 2021)."
Further, specifying that these operations are performed by “executing a neural network” is equivalent to merely apply a generic computer to perform the judicial exception, as analyzed below.
Additionally, should it be found that “query a fastener description repository” is not a mental process, it is also considered mere instructions to apply and well-understood, routine, conventional activity, as analyzed below.
selecting a first fastener and a second fastener; generating, using the neural network model, a merged fastener design based on the first fastener and the second fastener, wherein generating the merged fastener design comprises: executing a recurrent neural network (RNN) model of the neural network model to merge a first set of parameter values from the first fastener and a second set of parameter values from the second fastener to generate a set of estimated property values of the merged fastener design; executing a convolutional neural network (CNN) model of the neural network model using a first image of the first fastener and a second image of the second fastener to generate an image of the merged fastener design; and comparing the set of estimated property values of the merged fastener design with the image of the merged fastener design to detect convergence;
Choosing a pair of fasteners from a known set is a mental process equivalent to arbitrarily judging which fasteners to focus on. A merged design combining the features of both fasteners can be created by observing both, arbitrarily choosing a set of features from each, and creating, with a pencil and paper, a depiction of a new fastener with those properties. For example, a person could choose a Phillips head bolt that is 20mm in length and a hex head bolt that is 5mm in length and, after observing both and making feature choices, create a depiction of a new fastener that has a hex head and is 20mm in length. These properties can be depicted in a multitude of ways, such as text describing them (i.e. “hex head; 20mm”) or visually (i.e. drawing a hex head bolt that is 20mm long).
A person could also combine two fasteners purely visually by observing images of those fasteners and drawing, with a pencil and paper, a new fastener that combines the features of both images. For example, a person could observe an image of the 20mm Phillips head bolt and an image of the 5mm hex head bolt and draw a new bolt that is roughly 20mm long and has a hex head.
Finally, comparing the property values to the representation generated visually is a mental process equivalent to observing both drawing and judging how similar they are and if the features are consistent (i.e. are they both hex heads, are they both roughly 20mm, etc.) It should be further noted that such property values do not exclude imagery and visual properties themselves; i.e. properties such as size, shape, etc. are visual properties that are aspects of imagery. For example, a person could look at the characteristic shape of a Philips head (i.e. “+”) and visually compare that shape to the socket depicted in the compared fastener image to judge whether they are both Phillips heads.
Specifying that these operations are performed by an RNN and CNN is simply the act of instructing a computer to perform generic neural networks operations to carry out the mental process of combining the features of the fasteners, and therefore amounts to no more than mere instructions to apply a judicial exception using a generic computer.
Step 2A – Prong 2: Integrated into a Practical Solution?
Insignificant Extra-Solution Activity (MPEP 2106.05(g)) has found mere data gathering and post solution activity to be insignificant extra-solution activity.
Post-Solution Activity:
presenting the merged fastener design
This element merely acts on and presents the results of the previous abstract steps. A claim element that merely acts on and presents the results of a series of previous abstract steps is not indicative of integration into a practical solution nor evidence that the claim provides an inventive concept, as exemplified by ((MPEP 2106.05)(g)(Insignificant application) i. Cutting hair after first determining the hair style, In re Brown, 645 Fed. App'x 1014, 1016-1017 (Fed. Cir. 2016) and ii. Printing or downloading generated menus, Ameranth, 842 F.3d at 1241-42, 120 USPQ2d at 1854-55.)
Mere Instructions to Apply (MPEP 2106.05(f)) has found that merely applying a judicial exception such as an abstract idea, as by performing it on a computer, does not integrate the claim into a practical solution.
Mere Instructions to Apply:
executing a neural network model using features extracted at least from the initial set of fastener parameter values to query a fastener description repository; … executing a recurrent neural network (RNN) model of the neural network model … executing a convolutional neural network (CNN) model of the neural network model
Specifying that these operations are performed by a neural network at a high level of generality is simply the act of instructing a computer to perform generic neural network functions to carry out the mental process of searching for the desired fasteners and coming up with combinations of those fasteners, which is merely an instruction to apply a computer to the judicial exception. The claim only recites the idea of a solution or outcome, i.e. that the neural network is “executed,” the repository is “queried,” and the merged design is created without reciting how this execution, querying, or merging is actually accomplished. Further, the computer elements claimed are cited as merely generic tools to perform the operations.
The courts have found that such mere instructions to apply are not indicative of integration into a practical application nor recitation of significantly more than the judicial exception (MPEP 2106.05(f) “Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do "‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’". Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983”)
Additionally, querying a repository (i.e. a database) and retrieving data from it in a generic manner is also an example of using a computer merely as a tool to execute/apply the abstract idea. See MPEP § 2106.05(f)(2): “Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit) … TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea.”
Step 2B: Claim provides an Inventive Concept?
No, as discussed with respect to Step 2A, the additional limitations are mere data gathering or post solution activity (Insignificant Extra-Solution Activity), Well-Understood, Routine, Conventional Activity, or a general purpose computer and do not impose any meaningful limits on practicing the abstract idea and therefore the claim does not provide an inventive concept in Step 2B.
Insignificant Extra-Solution Activity (MPEP 2106.05(g)) has found mere data gathering and post solution activity to be insignificant extra-solution activity.
Post-Solution Activity:
presenting the merged fastener design
This element merely acts on and presents the results of the previous abstract steps. A claim element that merely acts on and presents the results of a series of previous abstract steps is not indicative of integration into a practical solution nor evidence that the claim provides an inventive concept, as exemplified by ((MPEP 2106.05)(g)(Insignificant application) i. Cutting hair after first determining the hair style, In re Brown, 645 Fed. App'x 1014, 1016-1017 (Fed. Cir. 2016) and ii. Printing or downloading generated menus, Ameranth, 842 F.3d at 1241-42, 120 USPQ2d at 1854-55.)
Mere Instructions to Apply (MPEP 2106.05(f)) has found that merely applying a judicial exception such as an abstract idea, as by performing it on a computer, does not integrate the claim into a practical solution.
Mere Instructions to Apply:
executing a neural network model using features extracted at least from the initial set of fastener parameter values to query a fastener description repository; … executing a recurrent neural network (RNN) model of the neural network model … executing a convolutional neural network (CNN) model of the neural network model
Specifying that these operations are performed by a neural network at a high level of generality is simply the act of instructing a computer to perform generic neural network functions to carry out the mental process of searching for the desired fasteners and coming up with combinations of those fasteners, which is merely an instruction to apply a computer to the judicial exception. The claim only recites the idea of a solution or outcome, i.e. that the neural network is “executed,” the repository is “queried,” and the merged design is created without reciting how this execution, querying, or merging is actually accomplished. Further, the computer elements claimed are cited as merely generic tools to perform the operations.
The courts have found that such mere instructions to apply are not indicative of integration into a practical application nor recitation of significantly more than the judicial exception (MPEP 2106.05(f) “Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do "‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’". Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983”)
Additionally, querying a repository (i.e. a database) and retrieving data from it in a generic manner is also an example of using a computer merely as a tool to execute/apply the abstract idea. See MPEP § 2106.05(f)(2): “Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit) … TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea.”
In addition, the following are also considered as well-understood, routine, and conventional activities, as discussed in MPEP § 2106.05(d):
“query a fastener description repository”
Querying and retrieving data from a database is equivalent to storing and retrieving information in memory as well as generic electronic recordkeeping (MPEP § 2106.05(d)(II) iii. Electronic recordkeeping, Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. 208, 225, 110 USPQ2d 1984 (2014) (creating and maintaining "shadow accounts"); Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755 (updating an activity log); iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;
“executing a convolutional neural network (CNN) model of the neural network model using a first image … and a second image … to generate an image of the merged … design;”
A Review of Multimodal Medical Image Fusion Techniques – ([Page 2 Par 1])
Deep learning for pixel-level image fusion: Recent advances and future prospects ([Page 161 Col 2 Par 7])
US 20190287215 A1 ([Abstract]
US 20190096046 A1 ([Abstract])
US 10467503 B1 ([Col 6 line 57-64])
“executing a recurrent neural network (RNN) model of the neural network model to merge a first set of parameter values … and a second set of parameter values … to generate a set of estimated property values”
US 20180206797 A1 ([Par 29, 40])
US 20180089888 A1 ([Par 50])
US 20180053108 A1 ([Par 38, 89, 108])
US 20170200065 A1 ([Par 76])
As per MPEP § 2106.05(d), an additional element that is “no more than well-understood, routine, conventional activities previously known to the industry, which is recited at a high level of generality,” does not integrate a judicial exception into a practical application, nor provide significantly more.
Moreover, Mere Instructions To Apply An Exception (MPEP 2106.05(f)) has found that simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. In light of this, the additional generic computer component elements of “executing a neural network model, a fastener description repository; executing a recurrent neural network (RNN) model of the neural network model; executing a convolutional neural network (CNN) model of the neural network model” are not sufficient to integrate a judicial exception into a practical application nor provide evidence of an inventive concept.
The claim is ineligible.
Claim 11 (Statutory Category – machine)
Yes, the claim recites a mental process, specifically:
MPEP 2106.04(a)(2)(Ill): “Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, Judgments, and opinions.”
Further, the MPEP recites “The courts do not distinguish between mental processes that are performed entirely in the human mind and mental processes that require a human to use a physical aid (e.g., pen and paper or a slide rule) to perform the claim limitation.”
obtaining an initial set of fastener parameter values for a fastener; executing a neural network model using features extracted at least from the initial set of fastener parameter values to query the fastener description repository; obtaining, from the fastener description repository, a set of possible matching fasteners; determining that a matching fastener does not exist;
Obtaining information in such a manner is a mental process equivalent to a person, such as a construction worker or carpenter, searching a hardware store for a certain type of fastener. The person may decide that, based on the needs of an ongoing project, they need Phillips head wood screws. Most hardware stores keep fasteners in organizers that group fasteners by type, such as intended use, head type, etc. that have visual keys to show where different types of hardware are stored. With this in mind such a worker would consult the key of the organizer, based on the desired fastener features, to find a collection of fasteners that meet their needs, such as a group of Phillips head wood screws of varying lengths. This kind of searching could also be done by observing a series of images of different fasteners, comparing the features of each, and judging if one or more meet the desired criteria. Determining that a matching fastener does not exist merely amounts to performing this search, having checked every option, and coming up empty handed. In other words, comparing each available fastener to the desired properties and judging, after exhausting all available fasteners, that none match these requirements.
See the July 2024 Patent Subject Matter Eligibility update: "A claim to “the collection of information from various sources (a Federal database, a State database, and a case worker) and understanding the meaning of that information (determining whether a person is receiving SSDI benefits and determining whether they are eligible for benefits under the law),” where “ `[t]hese steps can be performed by a human, using “observation, evaluation, judgment, [and] opinion,” because they involve making determinations and identifications, which are mental tasks humans routinely do,' ” and thus can practically be performed in the human mind, In re Killian, 45 F.4th 1373, 1379 (Fed. Cir. 2022). "
Also see in the July 2024 Patent Subject Matter Eligibility update: "Claims to “the use of an algorithm-generated content-based identifier to perform the claimed data-management functions,” which include limitations to “controlling access to data items,” “retrieving and delivering copies of data items,” and “marking copies of data items for deletion,” where the claims cover “a medley of mental processes that, taken together, amount only to a multistep mental process,” such that the steps can be practically performed in the human mind, PersonalWeb Techs. LLC v. Google LLC, 8 F.4th 1310, 1316-18 (Fed. Cir. 2021)."
Further, specifying that these operations are performed by “executing a neural network” is equivalent to merely apply a generic computer to perform the judicial exception, as analyzed below.
Additionally, should it be found that “query a fastener description repository” is not a mental process, it is also considered mere instructions to apply and well-understood, routine, conventional activity, as analyzed below.
selecting a first fastener and a second fastener; generating, using the neural network model, a merged fastener design based on the first fastener and the second fastener, wherein generating the merged fastener design comprises: executing a recurrent neural network (RNN) model of the neural network model to merge a first set of parameter values from the first fastener and a second set of parameter values from the second fastener to generate a set of estimated property values of the merged fastener design, wherein the set of estimated property values comprises one or more of a geometric property, a mechanical property, or a material composition property; executing a convolutional neural network (CNN) model of the neural network model using a first image of the first fastener and a second image of the second fastener to generate an image of the merged fastener design; comparing the set of estimated property values of the merged fastener design with the image of the merged fastener design to detect convergence; and
Choosing a pair of fasteners from a known set is a mental process equivalent to arbitrarily judging which fasteners to focus on. A merged design combining the features of both fasteners can be created by observing both, arbitrarily choosing a set of features from each, and creating, with a pencil and paper, a depiction of a new fastener with those properties. For example, a person could choose a Phillips head bolt that is 20mm in length and a hex head bolt that is 5mm in length and, after observing both and making feature choices, create a depiction of a new fastener that has a hex head and is 20mm in length. These properties can be depicted in a multitude of ways, such as text describing them (i.e. “hex head; 20mm”) or visually (i.e. drawing a hex head bolt that is 20mm long).
A person could also combine two fasteners purely visually by observing images of those fasteners and drawing, with a pencil and paper, a new fastener that combines the features of both images. For example, a person could observe an image of the 20mm Phillips head bolt and an image of the 5mm hex head bolt and draw a new bolt that is roughly 20mm long and has a hex head.
Finally, comparing the property values to the representation generated visually is a mental process equivalent to observing both drawing and judging how similar they are and if the features are consistent (i.e. are they both hex heads, are they both roughly 20mm, etc.) It should be further noted that such property values do not exclude imagery and visual properties themselves; i.e. properties such as size, shape, etc. are visual properties that are aspects of imagery. For example, a person could look at the characteristic shape of a Philips head (i.e. “+”) and visually compare that shape to the socket depicted in the compared fastener image to judge whether they are both Phillips heads.
Specifying that these operations are performed by an RNN and CNN is simply the act of instructing a computer to perform generic neural networks operations to carry out the mental process of combining the features of the fasteners, and therefore amounts to no more than mere instructions to apply a judicial exception using a generic computer.
Step 2A – Prong 2: Integrated into a Practical Solution?
Insignificant Extra-Solution Activity (MPEP 2106.05(g)) has found mere data gathering and post solution activity to be insignificant extra-solution activity.
Post-Solution Activity:
presenting the merged fastener design
This element merely acts on and presents the results of the previous abstract steps. A claim element that merely acts on and presents the results of a series of previous abstract steps is not indicative of integration into a practical solution nor evidence that the claim provides an inventive concept, as exemplified by ((MPEP 2106.05)(g)(Insignificant application) i. Cutting hair after first determining the hair style, In re Brown, 645 Fed. App'x 1014, 1016-1017 (Fed. Cir. 2016) and ii. Printing or downloading generated menus, Ameranth, 842 F.3d at 1241-42, 120 USPQ2d at 1854-55.)
Mere Instructions to Apply (MPEP 2106.05(f)) has found that merely applying a judicial exception such as an abstract idea, as by performing it on a computer, does not integrate the claim into a practical solution.
Mere Instructions to Apply:
executing a neural network model using features extracted at least from the initial set of fastener parameter values to query the fastener description repository; … executing a recurrent neural network (RNN) model of the neural network model … executing a convolutional neural network (CNN) model of the neural network model
Specifying that these operations are performed by a neural network at a high level of generality is simply the act of instructing a computer to perform generic neural network functions to carry out the mental process of searching for the desired fasteners and coming up with combinations of those fasteners, which is merely an instruction to apply a computer to the judicial exception. The claim only recites the idea of a solution or outcome, i.e. that the neural network is “executed,” the repository is “queried,” and the merged design is created without reciting how this execution, querying, or merging is actually accomplished. Further, the computer elements claimed are cited as merely generic tools to perform the operations.
The courts have found that such mere instructions to apply are not indicative of integration into a practical application nor recitation of significantly more than the judicial exception (MPEP 2106.05(f) “Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do "‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’". Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983”)
Additionally, querying a repository (i.e. a database) and retrieving data from it in a generic manner is also an example of using a computer merely as a tool to execute/apply the abstract idea. See MPEP § 2106.05(f)(2): “Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit) … TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea.”
Step 2B: Claim provides an Inventive Concept?
No, as discussed with respect to Step 2A, the additional limitations are mere data gathering or post solution activity (Insignificant Extra-Solution Activity), Well-Understood, Routine, Conventional Activity, or a general purpose computer and do not impose any meaningful limits on practicing the abstract idea and therefore the claim does not provide an inventive concept in Step 2B.
Insignificant Extra-Solution Activity (MPEP 2106.05(g)) has found mere data gathering and post solution activity to be insignificant extra-solution activity.
Post-Solution Activity:
presenting the merged fastener design
This element merely acts on and presents the results of the previous abstract steps. A claim element that merely acts on and presents the results of a series of previous abstract steps is not indicative of integration into a practical solution nor evidence that the claim provides an inventive concept, as exemplified by ((MPEP 2106.05)(g)(Insignificant application) i. Cutting hair after first determining the hair style, In re Brown, 645 Fed. App'x 1014, 1016-1017 (Fed. Cir. 2016) and ii. Printing or downloading generated menus, Ameranth, 842 F.3d at 1241-42, 120 USPQ2d at 1854-55.)
Mere Instructions to Apply (MPEP 2106.05(f)) has found that merely applying a judicial exception such as an abstract idea, as by performing it on a computer, does not integrate the claim into a practical solution.
Mere Instructions to Apply:
executing a neural network model using features extracted at least from the initial set of fastener parameter values to query the fastener description repository; … executing a recurrent neural network (RNN) model of the neural network model … executing a convolutional neural network (CNN) model of the neural network model
Specifying that these operations are performed by a neural network at a high level of generality is simply the act of instructing a computer to perform generic neural network functions to carry out the mental process of searching for the desired fasteners and coming up with combinations of those fasteners, which is merely an instruction to apply a computer to the judicial exception. The claim only recites the idea of a solution or outcome, i.e. that the neural network is “executed,” the repository is “queried,” and the merged design is created without reciting how this execution, querying, or merging is actually accomplished. Further, the computer elements claimed are cited as merely generic tools to perform the operations.
The courts have found that such mere instructions to apply are not indicative of integration into a practical application nor recitation of significantly more than the judicial exception (MPEP 2106.05(f) “Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do "‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’". Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983”)
Additionally, querying a database and retrieving data from it in a generic manner is also an example of using a computer merely as a tool to execute/apply the abstract idea. See MPEP § 2106.05(f)(2): “Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit) … TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea.”
In addition, the following are also considered as well-understood, routine, and conventional activities, as discussed in MPEP § 2106.05(d):
“query a fastener description repository”
Querying and retrieving data from a database is equivalent to storing and retrieving information in memory as well as generic electronic recordkeeping (MPEP § 2106.05(d)(II) iii. Electronic recordkeeping, Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. 208, 225, 110 USPQ2d 1984 (2014) (creating and maintaining "shadow accounts"); Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755 (updating an activity log); iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;
“executing a convolutional neural network (CNN) model of the neural network model using a first image … and a second image … to generate an image of the merged … design;”
A Review of Multimodal Medical Image Fusion Techniques – ([Page 2 Par 1])
Deep learning for pixel-level image fusion: Recent advances and future prospects ([Page 161 Col 2 Par 7])
US 20190287215 A1 ([Abstract]
US 20190096046 A1 ([Abstract])
US 10467503 B1 ([Col 6 line 57-64])
“executing a recurrent neural network (RNN) model of the neural network model to merge a first set of parameter values … and a second set of parameter values … to generate a set of estimated property values”
US 20180206797 A1 ([Par 29, 40])
US 20180089888 A1 ([Par 50])
US 20180053108 A1 ([Par 38, 89, 108])
US 20170200065 A1 ([Par 76])
As per MPEP § 2106.05(d), an additional element that is “no more than well-understood, routine, conventional activities previously known to the industry, which is recited at a high level of generality,” does not integrate a judicial exception into a practical application, nor provide significantly more.
Moreover, Mere Instructions To Apply An Exception (MPEP 2106.05(f)) has found that simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. In light of this, the additional generic computer component elements of “A system comprising: a fastener description repository comprising a plurality of fastener descriptions; and a computer processor configured to perform operations, the operations comprising: executing a neural network model; executing a recurrent neural network (RNN) model of the neural network model; executing a convolutional neural network (CNN) model of the neural network model” are not sufficient to integrate a judicial exception into a practical application nor provide evidence of an inventive concept.
The claim is ineligible.
Claim 19 The elements of claim 19 are substantially the same as those of claim 11. Therefore, the elements of claim 19 are rejected due to the same reasons as outlined above for claim 11.
Claim 2 recites “The method of claim 1, wherein the initial set of fastener parameter values comprises a plurality of partial parameter values, wherein executing the neural network model to query the fastener description repository comprises executing the RNN model to add an estimated set of parameter values to the initial set of fastener parameter values to create a revised set of parameter values; and wherein the method further comprises: querying the fastener description repository with the revised set of parameter values to obtain the set of possible matching fasteners.”
Revising parameters in such a manner is a mental process equivalent to adding more entries to a list of parameters, as by writing said list and additional entries with a pen and paper.
Specifying that the initial set of parameters is not a complete set of parameters merely clarifies the structure of the parameter set, and is therefore merely an extension of the mental process.
Specifying the neural network is an RNN merely clarifies the type of neural network used, and is therefore merely an extension of the mental process and mere instructions to apply the judicial exception.
Claim 3 recites “The method of claim 2, wherein executing the RNN model comprises: extracting a first set of features from the initial set of fastener parameter values; and extracting a second set of features from a context of the fastener, the context extracted from a design tool that designs an environment of the fastener; and executing the RNN model on the first set of features and the second set of features.”
Extracting features from parameters is a mental process equivalent to observing a set of parameters, for example coordinate information, and making a judgement about the features defined by those parameters. For example, a person could look at the coordinate set ((0,0), (1,0), (1,1), (0,1)) and determine that the shape defined by those coordinates is a square.
Specifying that these operations are performed by an RNN is simply the act of instructing a computer to perform generic neural network operations to carry out the mental process and therefore amounts to no more than mere instructions to apply a judicial exception using a generic computer.
See (MPEP 2106.05(f)(2)(i)) “A commonplace business method or mathematical algorithm being applied on a general purpose computer,” [Alice Corp. Pty. Ltd. V. CLS Bank Int’l, 573 U.S. 208, 223, 110 USPQ2d 1976, 1983 (2014); Gottschalk v. Benson, 409 U.S. 63, 64, 175 USPQ 673, 674 (1972); Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); ]
Further, should it be found that this extraction is not a mental process, it is also and example of mere data gathering.
“Extracting” this data in such a generic manner is equivalent to merely gathering data representative of features from the parameter values and context, and therefore amounts to no more than mere data gathering.
A claim element that amounts to merely gathering data is not indicative of integration into a
practical solution nor evidence that the claim provides an inventive concept or significantly more, as exemplified by ((MPEP 2106.05)(g)(Mere Data Gathering) i. Performing clinical tests on individuals to obtain input for an equation, In re Grams, 888 F.2d 835, 839-40; 12 USPQ2d 1824, 1827-28 (Fed. Cir. 1989); iv. Obtaining information about transactions using the Internet to verify credit card transactions, CyberSource v. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011);
Claim 4 recites “The method of claim 1, wherein the initial set of fastener parameter values comprises a submitted image of the fastener, and wherein executing the neural network model to query the fastener description repository comprises executing the CNN model to classify the submitted image based on a plurality of stored images in the fastener description repository.”
Specifying that the initial parameters comprise an image of a fastener and that the neural network is a CNN merely clarifies the structure of these elements, and is therefore merely an extension of the mental process and mere instructions to apply.
Classifying an image is a mental process that is possible to perform in the human mind. For example, a person could observe an image of a cat and judge that the animal in the image is a cat.
Claim 5 recites “The method of claim 1, wherein the initial set of fastener parameter values comprises a plurality of partial parameter values and a submitted image of the fastener, wherein the neural network model comprises a RNN model and a CNN model, wherein executing the RNN model adds an estimated set of parameter values to the initial set of fastener parameter values to create a revised set of parameter values for obtaining a first set of possible matching fasteners, wherein executing the CNN model comprises classifying the submitted image based on a plurality of stored images in the fastener description repository to obtain a second set of possible matching fasteners, and wherein the method further comprises comparing the first set of possible matching fasteners to the second set of possible matching fasteners to determine whether the matching fastener exists.”
Specifying that the initial parameters comprise an image of a fastener and that the initial set of parameters is not a complete set of parameters, merely clarifies the structure of these elements, and is therefore merely an extension of the mental process and mere instructions to apply.
Revising parameters in such a manner is a mental process equivalent to adding more entries to a list of parameters, as by writing said list and additional entries with a pen and paper.
Classifying an image is a mental process that is possible to perform in the human mind. For example, a person could observe an image of a cat and judge that the animal in the image is a cat.
Determining if a something exists in at least one of a plurality of sets of objects is a mental process equivalent to observing each object in each set and judging whether or not it matches the object being searched for.
Claim 6 recites “The method of claim 5, wherein comparing the first set of possible matching fasteners to the second set of possible matching fasteners comprises comparing alphanumeric identifiers assigned to the first set of possible matching fasteners and the second set of possible matching fasteners.”
Determining if a something exists in at least one of a plurality of sets of objects based on its label is a mental process equivalent to observing each object and its associated label in each set and judging whether or not the label matches the label for object being searched for.
Claim 21 recites wherein each of the plurality of stored images in the fastener description repository corresponds to an individual class.
Classifying an image is a mental process that is possible to perform in the human mind. For example, a person could observe an image of a cat and judge that the animal in the image is a cat. Specifying that only one class or category is associated with each image merely clarifies the content of the data contained within the repository.
Claim 12-15, and 22-23 The elements of claims 12-15, and 22-23 are substantially the same as those of claims 2-6 and 21. Therefore, the elements of claims 12-15, and 22-23 are rejected due to the same reasons as outlined above for claims 2-6 and 21.
Claims 24-28 The elements of claims 24-28 are substantially the same as those of claims 2-6. Therefore, the elements of claims 24-28 are rejected due to the same reasons as outlined above for claims 2-6.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 11-15, 19, and 21-28 are rejected under 35 U.S.C. 103 as being unpatentable over Automatic Detection and Identification of Fasteners with Simple Visual Calibration using Synthetic Data (hereinafter Noh) in view of Bass (WO 2009091584 A1) in further view of Zheng (US 10970768 B2).
Claim 1. Noh makes obvious A method comprising: obtaining an initial set of fastener parameter values for a fastener; executing a neural network model using features extracted at least from the initial set of fastener parameter values to ([Page 41 Col 2 Par 1] “The part identifier network outputs a separate set of scores for each main class to learn some shared features between object classes, similar to in [14]. In our case, there are a total of five diameters (M4-M12) and two main classes (bolt and nut), so the part identifier has a total of 10 outputs” [Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image… In model deployment, two networks are merged into a single end-to-end network to detect fastener regions pixel-wisely and identify both main class and diameter of fasteners”) obtaining, ([Page 41 Col 2 Par 1] “The part identifier network outputs a separate set of scores for each main class to learn some shared features between object classes, similar to in [14]. In our case, there are a total of five diameters (M4-M12) and two main classes (bolt and nut), so the part identifier has a total of 10 outputs” [Examiner’s note: the model outputs a list of scores for all 10 possible fastener configurations, i.e. potential matching fasteners, and labels the fastener based on which has the highest score] [Fig. 7] Shows images of a set of matched fasteners and labels/descriptions) ([Page 41 Col 1 Par 11] “We used Mask R-CNN [18]…”) a merged fastener design ([Page 40 Col 2 Par 2] “First, we created a total of 165 3D CAD models of bolt and nut for a model to learn general features from various kinds of fasteners. For bolts, we vary nominal diameter (M4, M6, M8, M10, and M12), head type (socket cap or hex), threaded type (fully threaded and partially threaded), and shaft length (10mm to 80mm at intervals of 5mm). We used hex nut, which is the most typical nut type, by chaning nominal diameters (M4, M6, M8, M10, and M12) … Using the virtual environment and domain randomization…”) ([Page 40 Col 2 Par 2] “First, we created a total of 165 3D CAD models of bolt and nut for a model to learn general features from various kinds of fasteners. For bolts, we vary nominal diameter (M4, M6, M8, M10, and M12), head type (socket cap or hex), threaded type (fully threaded and partially threaded), and shaft length (10mm to 80mm at intervals of 5mm). We used hex nut, which is the most typical nut type, by chaning nominal diameters (M4, M6, M8, M10, and M12) … Using the virtual environment and domain randomization…”) ([Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image… In model deployment, two networks are merged into a single end-to-end network to detect fastener regions pixel-wisely and identify both main class and diameter of fasteners”) to merge a first set of parameter values ([Page 40 Col 2 Par 2] “First, we created a total of 165 3D CAD models of bolt and nut for a model to learn general features from various kinds of fasteners. For bolts, we vary nominal diameter (M4, M6, M8, M10, and M12), head type (socket cap or hex), threaded type (fully threaded and partially threaded), and shaft length (10mm to 80mm at intervals of 5mm). We used hex nut, which is the most typical nut type, by chaning nominal diameters (M4, M6, M8, M10, and M12) … Using the virtual environment and domain randomization…” [Examiner’s note: the system merges the sets of parameter values (i.e. diameter, head type, length)]) executing a convolutional neural network (CNN) model of the neural network model ([Page 41 Col 1 Par 11] “We used Mask R-CNN [18] for the part detector network that is widely used for the instance segmentation task”)using a first image of the first fastener and a second image of the second fastener to generate an image of the merged fastener design; and ([Page 40 Col 2 Par 2 – Page 41 Col 1 Par 9] “First, we created a total of 165 3D CAD models of bolt and nut for a model to learn general features from various kinds of fasteners. For bolts, we vary nominal diameter (M4, M6, M8, M10, and M12), head type (socket cap or hex), threaded type (fully threaded and partially threaded), and shaft length (10mm to 80mm at intervals of 5mm). We used hex nut, which is the most typical nut type, by chaning nominal diameters (M4, M6, M8, M10, and M12) … Using the virtual environment and domain randomization [17], we generated the input image, instance segmentation mask, and reference image pairs by repeating the following steps: … 4) Put 1 to 5 random objects within an 8x6cm object region on the center of the floor without occlusion from CAD database using equal probability for the bolt and nut… 8) Randomly render the reference object with the same procedure in 5) and capture the camera scene for the reference image” … 9) While fixing the camera pose, repeat the procedure in 3) to 8) 25 times.” [Examiner’s note: the system generates several images of several different fasteners by merging/combining the properties of sets of parameters]) ([Page 40 Col 2 Par 2] “First, we created a total of 165 3D CAD models of bolt and nut for a model to learn general features from various kinds of fasteners. For bolts, we vary nominal diameter (M4, M6, M8, M10, and M12), head type (socket cap or hex), threaded type (fully threaded and partially threaded), and shaft length (10mm to 80mm at intervals of 5mm). We used hex nut, which is the most typical nut type, by chaning nominal diameters (M4, M6, M8, M10, and M12) … Using the virtual environment and domain randomization…”) ([Page 40 Col 2 Par 2 – Page 41 Col 1 Par 9] “Using the virtual environment and domain randomization [17], we generated the input image, instance segmentation mask, and reference image pairs by repeating the following steps: … 4) Put 1 to 5 random objects within an 8x6cm object region on the center of the floor without occlusion from CAD database using equal probability for the bolt and nut… 8) Randomly render the reference object with the same procedure in 5) and capture the camera scene for the reference image” … 9) While fixing the camera pose, repeat the procedure in 3) to 8) 25 times.”) ([Page 40 Col 2 Par 2 – Page 41 Col 1 Par 9] “Using the virtual environment and domain randomization [17], we generated the input image, instance segmentation mask, and reference image pairs by repeating the following steps: … 4) Put 1 to 5 random objects within an 8x6cm object region on the center of the floor without occlusion from CAD database using equal probability for the bolt and nut… 8) Randomly render the reference object with the same procedure in 5) and capture the camera scene for the reference image” … 9) While fixing the camera pose, repeat the procedure in 3) to 8) 25 times… The example of generated synthetic image pairs are shown in Figure 3”)
PNG
media_image1.png
555
635
media_image1.png
Greyscale
Noh does not explicitly teach query a fastener description repository; obtaining from the fastener description repository, data determining that a matching fastener does not exist; selecting a first fastener and a second fastener; performing an operation based on the first fastener and the second fastener, executing a recurrent neural network (RNN) model; parameter values from the first fastener; parameter values from the second fastener; comparing estimated properties with an image to detect convergence;
Bass makes obvious query a fastener description repository; obtaining from the fastener description repository, data ([Par 21] “The system itself relies upon a database to transform operator inputs, provided through an operator interface, to accurately identify an appropriate match or matches for the retail object in question. Typically, the input will be based upon easily identifiable traits found on the mast object, such as alphanumeric identifiers, lengths, head types, thread types, intended use(s) of the object (e.g., a wood fastener versus a concrete fastener) and the like.” [Par 24] “In some cases, the database output may include a number of items that represent a set of the closest or most appropriate matches for the user. In these instances, the database itself will be programmed to automatically search for and determine the occasions when multiple matches are appropriate, and the output from the database to the user will be provided accordingly.”) determining that a matching fastener does not exist; ([Par 36] “The information input into the system will enable the algorithm to select the appropriate fastener for identification. All selected fasteners can be located with the help of the identifier 22 for easy retrieval by the user. Preferably, based upon these inputs, a single fastener will be identified. When no single fastener is an exact match, a series of fastener possibilities may be identified. In the event an automated retrieval system is incorporated, the software can be programmed to afford the user with the option of retrieving all of the possible matches” [Examiner’s note: if a single exact match does not exist, a series (i.e. at least two) of fasteners are selected instead]) selecting a first fastener and a second fastener; performing an operation based on the first fastener and the second fastener, ([Par 36] “The information input into the system will enable the algorithm to select the appropriate fastener for identification. All selected fasteners can be located with the help of the identifier 22 for easy retrieval by the user. Preferably, based upon these inputs, a single fastener will be identified. When no single fastener is an exact match, a series of fastener possibilities may be identified. In the event an automated retrieval system is incorporated, the software can be programmed to afford the user with the option of retrieving all of the possible matches”) ([Par 36] “The information input into the system will enable the algorithm to select the appropriate fastener for identification. All selected fasteners can be located with the help of the identifier 22 for easy retrieval by the user. Preferably, based upon these inputs, a single fastener will be identified. When no single fastener is an exact match, a series of fastener possibilities may be identified. In the event an automated retrieval system is incorporated, the software can be programmed to afford the user with the option of retrieving all of the possible matches” [Par 20] “…easily identifiable traits found on the mast object, such as alphanumeric identifiers, lengths, head types, thread types, intended use(s) of the object (e.g., a wood fastener versus a concrete fastener) and the like.” [Par 21] “For example, with respect to a system for screw identification, some typical inputs required (and the corresponding variables contained in the database) might include: intended use (i.e., concrete, wood, steel), head style, material of construction, length, drive type (e.g., Phillips), and the like.” [Examiner’s note: each fastener has a set of parameter values like described in [Par 21])
Bass is analogous art because it is within the field of fastener database systems. It would have been obvious to one of ordinary skill in the art to combine it with Noh before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to make the fasteners that already exist easier to find. As noted by Bass, traditional methods for finding the exact fastener needed for a particular purpose can be extremely cumbersome due to the sheer quantity of different varieties of fasteners ([Par 4] “One illustrative example is the hardware field. In additional to numerous objects in the hardware field possessing minute yet important nuances, fasteners stand out as particularly cumbersome retail objects. Fasteners such as screws, nuts, bolts, washers, pins, hooks, or the like are provided with thousands of different specifications that are generally denoted in the metric system or British system”) To this end, Bass introduces a improved database system that allows the easy and rapid identification of fasteners that match desires specifications ([Par 7] “Therefore, a system that can provide better organizational structure and user- friendly operation would be a great improvement over the know prior know systems. Such a system would not only be valuable in the hardware field, but in any retail field where retail objects possess minute yet important nuances or functional variables.” [Par 8] “An interactive retail identification system for identifying a retail object based upon known or identified features of master object is disclosed, wherein the system utilizes a graphical user interface having a computer display screen for displaying a plurality of input fields related to a retail object and an input device for selecting variables based upon a physical inspection of a known master object having known or identifiable features”) Overall, one of ordinary skill in the art would have recognized that combining Noh with Bass would result in a system that was better capable of identifying fasteners that match given input.
The combination of Noh and Bass does not explicitly teach executing a recurrent neural network (RNN) and comparing estimated properties with an image to detect convergence;
Zheng makes obvious executing a recurrent neural network (RNN) and ([Col 18 line 52-56] “In another embodiment, the deep neural network 804 may comprise a hybrid neural network (termed a CRF-RNN) including a fully convolutional neural network and a recurrent network (RNN)”) comparing estimated properties with an image to detect convergence; ([Col 24 line 12-16] “At 1608, the methodology may calculate a visual similarity measure between the candidate product image portion and the input query image portion. At 1610, the methodology may output the visual similarity measure for use as a search result score for the candidate product.” [Col 19 line 56-58] “A visual search block 812 may calculate a visual similarity measure between input images, such as an image of a candidate product and the input query image.” [Examiner’s note: Based on the context of the claim language and its usage in [Par 45] of the specification, the term ‘convergence’ is interpreted as meaning that there is a certain level of similarity between two pieces of data. The “properties” described in the claim are interpreted as including visual properties, i.e. they would be represented in another image])
Zheng is analogous art because it is within the field of neural network data processing. It would have been obvious to combine Noh and Bass with Zheng before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to improve the fastener database searching system found in Bass. In particular, Zheng notes that traditional text-based searching systems struggle to effectively interpret visually-based search intent. ([Col 1 line 28-39] “Traditional searching is text-based rather than image-based or voice-based. Searching is overly time-consuming when too many irrelevant results must be presented, browsed, and rejected by a user. The technical limitations of conventional search tools make it difficult for a user to communicate search intent, for example by sharing photos of interesting products, to help start a search that may be refined by further user input, such as in a multi-turn dialog. As online searches balloon to billions of possible selectable products, comparison searching has become more important than ever, but current text-based solutions were not designed for this scale.”) In the context of the fastener search system, it would be appreciated that the ability to search for a fastener directly from an image of that fastener rather than laboriously describe it or enter details of the fastener would make the searching process significantly faster and easier. To this end, Zheng presents an artificial intelligence integrated search system that is significantly more intuitive than current methods, capable of accepting images as search input, and deeply integrated with product inventory systems ([Col 1 line 45-52] “ In one example, an intelligent personal assistant system includes scalable artificial intelligence (AI) … The system may leverage existing inventories and curated databases to provide intelligent, personalized answers in predictive turns of communication between a human user and an intelligent online personal assistant…. Machine learning components may continuously identify and learn from user intents so that user identity and understanding is enhanced over time. The user experience thus provided is inspiring, intuitive, unique, and may be focused on the usage” [Col 8 line 8-19] “A search component 220 is also included within the artificial intelligence framework 128… The back end unit may operate to manage item or product inventory and provide functions of searching against the inventory, optimizing towards a specific tuple of user intent and intent parameters. The search component 220 is designed to serve several billion queries per day globally against very large high quality inventories. The search component 220 can accommodate text, or Artificial Intelligence (AI) encoded voice and image inputs, and identify relevant inventory items to users based on explicit and derived query intents.”) Overall, one of ordinary skill in the art would have recognized that combining Zheng with Noh and Bass would result in a system that is significantly more intuitive to use and allows better database interaction by allowing direct database searching from images of a particular fastener rather than parameters describing said fastener.
Claim 11. The elements of claim 11 are substantially the same as those of claim 1. Therefore, the elements of claim 11 are rejected due to the same reasons as outlined above for claim 1. Further, Noh, Bass, and Zheng make obvious the additional elements of claim 11, particularly, Noh teaches A system comprisingset of fastener parameter values for a fastener; executing a neural network model using features extracted at least from the initial set of fastener parameter values to ([Page 41 Col 2 Par 1] “The part identifier network outputs a separate set of scores for each main class to learn some shared features between object classes, similar to in [14]. In our case, there are a total of five diameters (M4-M12) and two main classes (bolt and nut), so the part identifier has a total of 10 outputs” [Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image… In model deployment, two networks are merged into a single end-to-end network to detect fastener regions pixel-wisely and identify both main class and diameter of fasteners”) obtaining, ([Page 41 Col 2 Par 1] “The part identifier network outputs a separate set of scores for each main class to learn some shared features between object classes, similar to in [14]. In our case, there are a total of five diameters (M4-M12) and two main classes (bolt and nut), so the part identifier has a total of 10 outputs” [Examiner’s note: the model outputs a list of scores for all 10 possible fastener configurations, i.e. potential matching fasteners, and labels the fastener based on which has the highest score] [Fig. 7] Shows images of a set of matched fasteners and labels/descriptions) generate a set of estimated property values of the merged fastener design, wherein the set of estimated property values comprises one or more of a geometric property, a mechanical property, or a material composition property; ([Page 40 Col 2 Par 2] “First, we created a total of 165 3D CAD models of bolt and nut for a model to learn general features from various kinds of fasteners. For bolts, we vary nominal diameter (M4, M6, M8, M10, and M12), head type (socket cap or hex), threaded type (fully threaded and partially threaded), and shaft length (10mm to 80mm at intervals of 5mm). We used hex nut, which is the most typical nut type, by chaning nominal diameters (M4, M6, M8, M10, and M12) … Using the virtual environment and domain randomization…” [Examiner’s note: the system merges the sets of parameter values (i.e. diameter, head type, length)])
Noh does not explicitly teach a fastener description repository comprising a plurality of fastener descriptions; and a computer processor configured to perform operations; query a fastener description repository; obtaining from the fastener description repository, data
Bass makes obvious a fastener description repository comprising a plurality of fastener descriptions; and a computer processor configured to perform operations; query a fastener description repository; obtaining from the fastener description repository, data ([Par 21] “The system itself relies upon a database to transform operator inputs, provided through an operator interface, to accurately identify an appropriate match or matches for the retail object in question. Typically, the input will be based upon easily identifiable traits found on the mast object, such as alphanumeric identifiers, lengths, head types, thread types, intended use(s) of the object (e.g., a wood fastener versus a concrete fastener) and the like.” [Par 24] “In some cases, the database output may include a number of items that represent a set of the closest or most appropriate matches for the user. In these instances, the database itself will be programmed to automatically search for and determine the occasions when multiple matches are appropriate, and the output from the database to the user will be provided accordingly.” [Par 31] “Notably, keeping in mind that a monitor, CRT or video system is normally included in most computer systems, the output of the present invention includes a specially designed identifier 22 that highlights or otherwise communicates possible matches for the operator's further visual examination and/or use” [Par 22] “For example, with respect to a fastener identification system, the most practical implementation scheme may be through the use of a common, over-the-counter personal computer,”)
Claim 19. The elements of claim 19 are substantially the same as those of claim 11. Therefore, the elements of claim 19 are rejected due to the same reasons as outlined above for claim 11. Further, the combination of Noh, Bass, and Zheng makes obvious the additional elements of claim 19, particularly Bass makes obvious A non-transitory computer readable medium comprising computer readable program code for causing a computer system to perform operations, the operations comprising: ([Par 22] “For example, with respect to a fastener identification system, the most practical implementation scheme may be through the use of a common, over-the-counter personal computer,”)
Claim 2. Noh teaches wherein the initial set of fastener parameter values comprises a plurality of partial parameter values, wherein executing the neural network model to(([Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image…Then, each instance mask is concatenated with a reference image that was taken and stored for visual calibration in that camera pose. Lastly, the part identifier network takes each single instance mask one-by-one, along with the reference image, and predicts the nominal diameter for each detected instance.” [Page 40 Col 2 Par 2] “Using the virtual environment and domain randomization [17], we generated the input image, instance segmentation mask, and reference image pairs by repeating the following steps:” [Examiner’s note: the instance segmentation masks are interpreted as an example of partial parameter values as they only describe which parts of the image are to be obscured/occluded and do not contain the full detail of the input image, i.e. only part of the parameters])) and wherein the method further comprises: ([Page 41 Col 1 Par 10] “Then, each instance mask is concatenated with a reference image that was taken and stored for visual calibration in that camera pose. Lastly, the part identifier network takes each single instance mask one-by-one, along with the reference image, and predicts the nominal diameter for each detected instance.”) to obtain the set of possible matching fasteners. ([Page 41 Col 2 Par 1] “The part identifier network outputs a separate set of scores for each main class to learn some shared features between object classes, similar to in [14]. In our case, there are a total of five diameters (M4-M12) and two main classes (bolt and nut), so the part identifier has a total of 10 outputs” [Fig. 7] Shows images of a set of matched fasteners and labels/descriptions)
Bass makes obvious using the system to query the fastener description repository; ([Par 21] “The system itself relies upon a database to transform operator inputs, provided through an operator interface, to accurately identify an appropriate match or matches for the retail object in question. Typically, the input will be based upon easily identifiable traits found on the mast object, such as alphanumeric identifiers, lengths, head types, thread types, intended use(s) of the object (e.g., a wood fastener versus a concrete fastener) and the like.” [Par 24] “In some cases, the database output may include a number of items that represent a set of the closest or most appropriate matches for the user. In these instances, the database itself will be programmed to automatically search for and determine the occasions when multiple matches are appropriate, and the output from the database to the user will be provided accordingly.” [Par 31] “Notably, keeping in mind that a monitor, CRT or video system is normally included in most computer systems, the output of the present invention includes a specially designed identifier 22 that highlights or otherwise communicates possible matches for the operator's further visual examination and/or use”)
Zheng makes obvious executing the RNN; ([Col 18 line 52-56] “In another embodiment, the deep neural network 804 may comprise a hybrid neural network (termed a CRF-RNN) including a fully convolutional neural network and a recurrent network (RNN)”)
Claim 3. Noh teaches wherein executing the RNN model comprises: extracting a first set of features from the initial set of fastener parameter values; ([Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image…”) and extracting a second set of features from a context of the fastener, ([Page 41 Col 1 Par 10] “Then, each instance mask is concatenated with a reference image that was taken and stored for visual calibration in that camera pose. Lastly, the part identifier network takes each single instance mask one-by-one, along with the reference image, and predicts the nominal diameter for each detected instance. Thus, the model can infer the actual scale of detected images by referring the pixel-level scale of reference objects when it predicts the diameter…Note that for each new camera pose, visual calibration can be conducted simply by just capturing a reference object (coin) once, which can be extended to mobile part identification systems.”) the context extracted from a design tool that designs an environment of the fastener; ([Page 40 Col 1 Par 1] “We propose a synthetic data generation method and deep learning models to detect and identify multiple fasteners automatically with simple visual calibration. The pipeline of our proposed system is depicted in Figure 1. To reduce the cost for data acquisition and enable fast model retraining under rapid production cycles, we first generated a synthetic dataset (Figure 1a) using a virtual environment.” [Page 40 Col 2 Par 2 – Page 41 Col 1 Par 8] “Using the virtual environment and domain randomization [17], we generated the input image, instance segmentation mask, and reference image pairs by repeating the following steps: … 1) Randomize the camera’s field of view … 3) Set the intensity and color of five lights with a random value while keeping the color of one of light as white. 4) Put 1 to 5 random objects within an 8x6cm object region on the center of the floor … 6) Generate an input image for the model and a corresponding pixel-wise instance segmentation mask in that scene. 7) Remove all objects then put a reference object(coin) on the random position in a 3x3 cm region on the floor 8) Randomly render the reference object with the same procedure in 5) and capture the camera scene for the reference image”) and executing the RNN model on the first set of features and the second set of features. ([Page 41 Col 1 Par 10] “First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image. Then, each instance mask is concatenated with a reference image that was taken and stored for visual calibration in that camera pose. Lastly, the part identifier network takes each single instance mask one-by-one, along with the reference image, and predicts the nominal diameter for each detected instance. Thus, the model can infer the actual scale of detected images by referring the pixel-level scale of reference objects when it predicts the diameter… In model deployment, two networks are merged into a single end-to-end network to detect fastener regions pixel-wisely and identify both main class and diameter of fasteners” [Page 41 Col 1 Par 11] “We used Mask R-CNN [18] for the part detector network that is widely used for the instance segmentation task. As a backbone, ResNet-50 [19] with Feature Pyramid Network [20] was used. For part identifier network, ResNet-32 [19] was used for feature extractor”)
Zheng makes obvious the RNN model([Col 18 line 52-56] “In another embodiment, the deep neural network 804 may comprise a hybrid neural network (termed a CRF-RNN) including a fully convolutional neural network and a recurrent network (RNN)”)
Claim 4. Noh teaches wherein the initial set of fastener parameter values comprises a submitted image of the fastener, ([Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image…”) and wherein executing the neural network model ([Page 41 Col 2 Par 1] “The part identifier network outputs a separate set of scores for each main class to learn some shared features between object classes, similar to in [14]. In our case, there are a total of five diameters (M4-M12) and two main classes (bolt and nut), so the part identifier has a total of 10 outputs” [Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image… In model deployment, two networks are merged into a single end-to-end network to detect fastener regions pixel-wisely and identify both main class and diameter of fasteners”) ([Page 41 Col 1 Par 11] “We used Mask R-CNN [18] for the part detector network that is widely used for the instance segmentation task. As a backbone, ResNet-50 [19] with Feature Pyramid Network [20] was used.”) to classify the submitted image based on a plurality of stored images in the ([Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image…” [Page 40 Col 2 Par 2 – Page 41 Col 1 Par 9] “Using the virtual environment and domain randomization [17], we generated the input image, instance segmentation mask, and reference image pairs by repeating the following steps: … As a result, a total of 35,000 training and 5,000 validation set were created, and data generation took approximately 12 hours. The example of generated synthetic image pairs are shown in Figure 3.” [Fig. 3] Shows stored images)
Bass makes obvious using the system to query the fastener description repository; data from the fastener description repository ([Par 21] “The system itself relies upon a database to transform operator inputs, provided through an operator interface, to accurately identify an appropriate match or matches for the retail object in question. Typically, the input will be based upon easily identifiable traits found on the mast object, such as alphanumeric identifiers, lengths, head types, thread types, intended use(s) of the object (e.g., a wood fastener versus a concrete fastener) and the like.” [Par 24] “In some cases, the database output may include a number of items that represent a set of the closest or most appropriate matches for the user. In these instances, the database itself will be programmed to automatically search for and determine the occasions when multiple matches are appropriate, and the output from the database to the user will be provided accordingly.” [Par 31] “Notably, keeping in mind that a monitor, CRT or video system is normally included in most computer systems, the output of the present invention includes a specially designed identifier 22 that highlights or otherwise communicates possible matches for the operator's further visual examination and/or use”)
Claim 5. Noh teaches wherein the initial set of fastener parameter values comprises a plurality of partial parameter values and a submitted image of the fastener, ([Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image… Then, each instance mask is concatenated with a reference image that was taken and stored for visual calibration in that camera pose. Lastly, the part identifier network takes each single instance mask one-by-one, along with the reference image, and predicts the nominal diameter for each detected instance.” [Page 40 Col 2 Par 2] “Using the virtual environment and domain randomization [17], we generated the input image, instance segmentation mask, and reference image pairs by repeating the following steps:”) wherein executing the ([Page 41 Col 1 Par 10] “Then, each instance mask is concatenated with a reference image that was taken and stored for visual calibration in that camera pose. Lastly, the part identifier network takes each single instance mask one-by-one, along with the reference image, and predicts the nominal diameter for each detected instance.”) for obtaining a first set of possible matching fasteners, ([Page 41 Col 2 Par 1] “The part identifier network outputs a separate set of scores for each main class to learn some shared features between object classes, similar to in [14]. In our case, there are a total of five diameters (M4-M12) and two main classes (bolt and nut), so the part identifier has a total of 10 outputs” [Fig. 7] Shows images of a set of matched fasteners and labels/descriptions) wherein executing the CNN model ([Page 41 Col 1 Par 11] “We used Mask R-CNN [18] for the part detector network that is widely used for the instance segmentation task. As a backbone, ResNet-50 [19] with Feature Pyramid Network [20] was used.”) comprises classifying the submitted image based on a plurality of stored images ([Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image…” [Page 40 Col 2 Par 2 – Page 41 Col 1 Par 9] “Using the virtual environment and domain randomization [17], we generated the input image, instance segmentation mask, and reference image pairs by repeating the following steps: … As a result, a total of 35,000 training and 5,000 validation set were created, and data generation took approximately 12 hours. The example of generated synthetic image pairs are shown in Figure 3.” [Fig. 3] Shows stored images) and wherein the method further comprises ([Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image…” [Page 41 Col 1 Par 10] “Then, each instance mask is concatenated with a reference image that was taken and stored for visual calibration in that camera pose. Lastly, the part identifier network takes each single instance mask one-by-one, along with the reference image, and predicts the nominal diameter for each detected instance.” [Page 41 Col 2 Par 1] “The part identifier network outputs a separate set of scores for each main class to learn some shared features between object classes, similar to in [14]. In our case, there are a total of five diameters (M4-M12) and two main classes (bolt and nut), so the part identifier has a total of 10 outputs” [Fig. 6a] Shows a fully matched final fastener [Examiner’s note: the part detector model creates an set of possible matches by narrowing the total possibilities by type (i.e. nut or bolt), then the part identifier model creates a set of possible matches narrowed by potential diameters. Together, the two sets determine a final matching fastener])
Bass makes obvious ([Par 21] “The system itself relies upon a database to transform operator inputs, provided through an operator interface, to accurately identify an appropriate match or matches for the retail object in question. Typically, the input will be based upon easily identifiable traits found on the mast object, such as alphanumeric identifiers, lengths, head types, thread types, intended use(s) of the object (e.g., a wood fastener versus a concrete fastener) and the like.” [Par 24] “In some cases, the database output may include a number of items that represent a set of the closest or most appropriate matches for the user. In these instances, the database itself will be programmed to automatically search for and determine the occasions when multiple matches are appropriate, and the output from the database to the user will be provided accordingly.” [Par 23] “Functionally, the database compares and cross-references a plurality of known variables against at least one input provided by the user in a manner that is well known to those skilled in the arts of software and computer-related inventions.”) wherein the method further comprises comparing matches to determine whether the matching fastener exists ([Par 24] “In some cases, the database output may include a number of items that represent a set of the closest or most appropriate matches for the user. In these instances, the database itself will be programmed to automatically search for and determine the occasions when multiple matches are appropriate, and the output from the database to the user will be provided accordingly.” [Par 23] “Functionally, the database compares and cross-references a plurality of known variables against at least one input provided by the user in a manner that is well known to those skilled in the arts of software and computer-related inventions.” [Par 36] “The information input into the system will enable the algorithm to select the appropriate fastener for identification. All selected fasteners can be located with the help of the identifier 22 for easy retrieval by the user. Preferably, based upon these inputs, a single fastener will be identified. When no single fastener is an exact match … “)
Zheng makes obvious the RNN model; ([Col 18 line 52-56] “In another embodiment, the deep neural network 804 may comprise a hybrid neural network (termed a CRF-RNN) including a fully convolutional neural network and a recurrent network (RNN)”)
Claim 6. Noh teaches wherein ([Page 41 Col 1 Par 10] “Then, each instance mask is concatenated with a reference image that was taken and stored for visual calibration in that camera pose. Lastly, the part identifier network takes each single instance mask one-by-one, along with the reference image, and predicts the nominal diameter for each detected instance.” [Page 41 Col 2 Par 1] “The part identifier network outputs a separate set of scores for each main class to learn some shared features between object classes, similar to in [14]. In our case, there are a total of five diameters (M4-M12) and two main classes (bolt and nut), so the part identifier has a total of 10 outputs” [Fig. 6a] Shows a fully matched final fastener]) to the second set of possible matching fasteners ([Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image…”) ([Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image…”) and the second set of possible matching fasteners. ([Page 41 Col 1 Par 10] “Then, each instance mask is concatenated with a reference image that was taken and stored for visual calibration in that camera pose. Lastly, the part identifier network takes each single instance mask one-by-one, along with the reference image, and predicts the nominal diameter for each detected instance.”)
Bass makes obvious wherein comparing matches comprises comparing alphanumeric identifiers assigned to possible matches ([Par 21] “The system itself relies upon a database to transform operator inputs, provided through an operator interface, to accurately identify an appropriate match or matches for the retail object in question. Typically, the input will be based upon easily identifiable traits found on the mast object, such as alphanumeric identifiers, lengths, head types, thread types, intended use(s) of the object (e.g., a wood fastener versus a concrete fastener) and the like.” [Par 23] “Functionally, the database compares and cross-references a plurality of known variables against at least one input provided by the user in a manner that is well known to those skilled in the arts of software and computer-related inventions”)
Claim 21. Noh teaches wherein each of the plurality of stored images ([Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image…” [Page 40 Col 2 Par 2 – Page 41 Col 1 Par 9] “Using the virtual environment and domain randomization [17], we generated the input image, instance segmentation mask, and reference image pairs by repeating the following steps: … As a result, a total of 35,000 training and 5,000 validation set were created, and data generation took approximately 12 hours. The example of generated synthetic image pairs are shown in Figure 3.” [Fig. 3] Shows stored images)
Bass makes obvious ([Par 21] “The system itself relies upon a database to transform operator inputs, provided through an operator interface, to accurately identify an appropriate match or matches for the retail object in question. Typically, the input will be based upon easily identifiable traits found on the mast object, such as alphanumeric identifiers, lengths, head types, thread types, intended use(s) of the object (e.g., a wood fastener versus a concrete fastener) and the like.” [Par 24] “In some cases, the database output may include a number of items that represent a set of the closest or most appropriate matches for the user. In these instances, the database itself will be programmed to automatically search for and determine the occasions when multiple matches are appropriate, and the output from the database to the user will be provided accordingly.” [Par 23] “Functionally, the database compares and cross-references a plurality of known variables against at least one input provided by the user in a manner that is well known to those skilled in the arts of software and computer-related inventions.”)
Zheng makes obvious wherein each image in the system corresponds to an individual category ([Col 19 line 23-28] “A leaf category prediction block 806 may help determine to which particular potentially defined category or subcategory in an electronic marketplace a given image is related (e.g., “men's dress shoes”, “dome camping tent”) based on the visual text content provided. Use of predicted categories may sharply reduce the possible search space in an electronic marketplace that may have a very large overall number of product listings available. This search space reduction may increase both the speed of a search and the relevance of the search results found.”)
Claim 12-15, and 22-23. The elements of claims 12-15, and 22-23 are substantially the same as those of claims 2-6 and 21. Therefore, the elements of claims 12-15, and 22-23 are rejected due to the same reasons as outlined above for claims 2-6 and 21.
Further, Noh, Bass, and Zheng make obvious the additional elements of claim 11 as inherited by claims 12-15, and 22-23, particularly, Noh teaches A system comprising([Page 41 Col 2 Par 1] “The part identifier network outputs a separate set of scores for each main class to learn some shared features between object classes, similar to in [14]. In our case, there are a total of five diameters (M4-M12) and two main classes (bolt and nut), so the part identifier has a total of 10 outputs” [Page 41 Col 1 Par 10] “Our proposed system for the detection and identification of fasteners consists of two deep learning models, part detector and part identifier, as shown in Figure 2. First, the part detector network predicts pixel-wise instance masks and its main class (bolt and nuts) from the input image… In model deployment, two networks are merged into a single end-to-end network to detect fastener regions pixel-wisely and identify both main class and diameter of fasteners”) obtaining, ([Page 41 Col 2 Par 1] “The part identifier network outputs a separate set of scores for each main class to learn some shared features between object classes, similar to in [14]. In our case, there are a total of five diameters (M4-M12) and two main classes (bolt and nut), so the part identifier has a total of 10 outputs” [Examiner’s note: the model outputs a list of scores for all 10 possible fastener configurations, i.e. potential matching fasteners, and labels the fastener based on which has the highest score] [Fig. 7] Shows images of a set of matched fasteners and labels/descriptions) generate a set of estimated property values of the merged fastener design, wherein the set of estimated property values comprises one or more of a geometric property, a mechanical property, or a material composition property; ([Page 40 Col 2 Par 2] “First, we created a total of 165 3D CAD models of bolt and nut for a model to learn general features from various kinds of fasteners. For bolts, we vary nominal diameter (M4, M6, M8, M10, and M12), head type (socket cap or hex), threaded type (fully threaded and partially threaded), and shaft length (10mm to 80mm at intervals of 5mm). We used hex nut, which is the most typical nut type, by chaning nominal diameters (M4, M6, M8, M10, and M12) … Using the virtual environment and domain randomization…” [Examiner’s note: the system merges the sets of parameter values (i.e. diameter, head type, length)])
Noh does not explicitly teach a fastener description repository comprising a plurality of fastener descriptions; and a computer processor configured to perform operations; query a fastener description repository; obtaining from the fastener description repository, data
Bass makes obvious a fastener description repository comprising a plurality of fastener descriptions; and a computer processor configured to perform operations; query a fastener description repository; obtaining from the fastener description repository, data ([Par 21] “The system itself relies upon a database to transform operator inputs, provided through an operator interface, to accurately identify an appropriate match or matches for the retail object in question. Typically, the input will be based upon easily identifiable traits found on the mast object, such as alphanumeric identifiers, lengths, head types, thread types, intended use(s) of the object (e.g., a wood fastener versus a concrete fastener) and the like.” [Par 24] “In some cases, the database output may include a number of items that represent a set of the closest or most appropriate matches for the user. In these instances, the database itself will be programmed to automatically search for and determine the occasions when multiple matches are appropriate, and the output from the database to the user will be provided accordingly.” [Par 31] “Notably, keeping in mind that a monitor, CRT or video system is normally included in most computer systems, the output of the present invention includes a specially designed identifier 22 that highlights or otherwise communicates possible matches for the operator's further visual examination and/or use” [Par 22] “For example, with respect to a fastener identification system, the most practical implementation scheme may be through the use of a common, over-the-counter personal computer,”)
Claims 24-28. The elements of claims 24-28 are substantially the same as those of claims 2-6. Therefore, the elements of claims 24-28 are rejected due to the same reasons as outlined above for claims 2-6. Further, the combination of Noh, Bass, and Zheng makes obvious the additional elements of claim 19 as inherited by claims 24-28. Particularly, Noh teaches ([Page 40 Col 2 Par 2] “First, we created a total of 165 3D CAD models of bolt and nut for a model to learn general features from various kinds of fasteners. For bolts, we vary nominal diameter (M4, M6, M8, M10, and M12), head type (socket cap or hex), threaded type (fully threaded and partially threaded), and shaft length (10mm to 80mm at intervals of 5mm). We used hex nut, which is the most typical nut type, by chaning nominal diameters (M4, M6, M8, M10, and M12) … Using the virtual environment and domain randomization…” [Examiner’s note: the system merges the sets of parameter values (i.e. diameter, head type, length)])
Noh does not explicitly teach A non-transitory computer readable medium comprising computer readable program code for causing a computer system to perform operations, the operations comprising: …
Bass makes obvious A non-transitory computer readable medium comprising computer readable program code for causing a computer system to perform operations, the operations comprising: ([Par 22] “For example, with respect to a fastener identification system, the most practical implementation scheme may be through the use of a common, over-the-counter personal computer,”)
Conclusion
Prior art of note that is not relied upon is made of record below:
Learning Two-Branch Neural Networks for Image-Text Matching Tasks
Multimodal Convolutional Neural Networks for Matching Image and Sentence
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael P Mirabito whose telephone number is (703)756-1494. The examiner can normally be reached M-F 10:30 am - 6:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emerson Puente can be reached at (571) 272-3652. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.P.M./ Examiner, Art Unit 2187
/EMERSON C PUENTE/ Supervisory Patent Examiner, Art Unit 2187