Codebase list certgraph / d59f704
New upstream version 20180911 Sophie Brun 3 years ago
22 changed file(s) with 3023 addition(s) and 0 deletion(s). Raw diff Collapse all Expand all
0 go.sum
1 certgraph
2 build/
3 *.json
0 GNU GENERAL PUBLIC LICENSE
1 Version 2, June 1991
2
3 Copyright (C) 1989, 1991 Free Software Foundation, Inc., <http://fsf.org/>
4 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
5 Everyone is permitted to copy and distribute verbatim copies
6 of this license document, but changing it is not allowed.
7
8 Preamble
9
10 The licenses for most software are designed to take away your
11 freedom to share and change it. By contrast, the GNU General Public
12 License is intended to guarantee your freedom to share and change free
13 software--to make sure the software is free for all its users. This
14 General Public License applies to most of the Free Software
15 Foundation's software and to any other program whose authors commit to
16 using it. (Some other Free Software Foundation software is covered by
17 the GNU Lesser General Public License instead.) You can apply it to
18 your programs, too.
19
20 When we speak of free software, we are referring to freedom, not
21 price. Our General Public Licenses are designed to make sure that you
22 have the freedom to distribute copies of free software (and charge for
23 this service if you wish), that you receive source code or can get it
24 if you want it, that you can change the software or use pieces of it
25 in new free programs; and that you know you can do these things.
26
27 To protect your rights, we need to make restrictions that forbid
28 anyone to deny you these rights or to ask you to surrender the rights.
29 These restrictions translate to certain responsibilities for you if you
30 distribute copies of the software, or if you modify it.
31
32 For example, if you distribute copies of such a program, whether
33 gratis or for a fee, you must give the recipients all the rights that
34 you have. You must make sure that they, too, receive or can get the
35 source code. And you must show them these terms so they know their
36 rights.
37
38 We protect your rights with two steps: (1) copyright the software, and
39 (2) offer you this license which gives you legal permission to copy,
40 distribute and/or modify the software.
41
42 Also, for each author's protection and ours, we want to make certain
43 that everyone understands that there is no warranty for this free
44 software. If the software is modified by someone else and passed on, we
45 want its recipients to know that what they have is not the original, so
46 that any problems introduced by others will not reflect on the original
47 authors' reputations.
48
49 Finally, any free program is threatened constantly by software
50 patents. We wish to avoid the danger that redistributors of a free
51 program will individually obtain patent licenses, in effect making the
52 program proprietary. To prevent this, we have made it clear that any
53 patent must be licensed for everyone's free use or not licensed at all.
54
55 The precise terms and conditions for copying, distribution and
56 modification follow.
57
58 GNU GENERAL PUBLIC LICENSE
59 TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
60
61 0. This License applies to any program or other work which contains
62 a notice placed by the copyright holder saying it may be distributed
63 under the terms of this General Public License. The "Program", below,
64 refers to any such program or work, and a "work based on the Program"
65 means either the Program or any derivative work under copyright law:
66 that is to say, a work containing the Program or a portion of it,
67 either verbatim or with modifications and/or translated into another
68 language. (Hereinafter, translation is included without limitation in
69 the term "modification".) Each licensee is addressed as "you".
70
71 Activities other than copying, distribution and modification are not
72 covered by this License; they are outside its scope. The act of
73 running the Program is not restricted, and the output from the Program
74 is covered only if its contents constitute a work based on the
75 Program (independent of having been made by running the Program).
76 Whether that is true depends on what the Program does.
77
78 1. You may copy and distribute verbatim copies of the Program's
79 source code as you receive it, in any medium, provided that you
80 conspicuously and appropriately publish on each copy an appropriate
81 copyright notice and disclaimer of warranty; keep intact all the
82 notices that refer to this License and to the absence of any warranty;
83 and give any other recipients of the Program a copy of this License
84 along with the Program.
85
86 You may charge a fee for the physical act of transferring a copy, and
87 you may at your option offer warranty protection in exchange for a fee.
88
89 2. You may modify your copy or copies of the Program or any portion
90 of it, thus forming a work based on the Program, and copy and
91 distribute such modifications or work under the terms of Section 1
92 above, provided that you also meet all of these conditions:
93
94 a) You must cause the modified files to carry prominent notices
95 stating that you changed the files and the date of any change.
96
97 b) You must cause any work that you distribute or publish, that in
98 whole or in part contains or is derived from the Program or any
99 part thereof, to be licensed as a whole at no charge to all third
100 parties under the terms of this License.
101
102 c) If the modified program normally reads commands interactively
103 when run, you must cause it, when started running for such
104 interactive use in the most ordinary way, to print or display an
105 announcement including an appropriate copyright notice and a
106 notice that there is no warranty (or else, saying that you provide
107 a warranty) and that users may redistribute the program under
108 these conditions, and telling the user how to view a copy of this
109 License. (Exception: if the Program itself is interactive but
110 does not normally print such an announcement, your work based on
111 the Program is not required to print an announcement.)
112
113 These requirements apply to the modified work as a whole. If
114 identifiable sections of that work are not derived from the Program,
115 and can be reasonably considered independent and separate works in
116 themselves, then this License, and its terms, do not apply to those
117 sections when you distribute them as separate works. But when you
118 distribute the same sections as part of a whole which is a work based
119 on the Program, the distribution of the whole must be on the terms of
120 this License, whose permissions for other licensees extend to the
121 entire whole, and thus to each and every part regardless of who wrote it.
122
123 Thus, it is not the intent of this section to claim rights or contest
124 your rights to work written entirely by you; rather, the intent is to
125 exercise the right to control the distribution of derivative or
126 collective works based on the Program.
127
128 In addition, mere aggregation of another work not based on the Program
129 with the Program (or with a work based on the Program) on a volume of
130 a storage or distribution medium does not bring the other work under
131 the scope of this License.
132
133 3. You may copy and distribute the Program (or a work based on it,
134 under Section 2) in object code or executable form under the terms of
135 Sections 1 and 2 above provided that you also do one of the following:
136
137 a) Accompany it with the complete corresponding machine-readable
138 source code, which must be distributed under the terms of Sections
139 1 and 2 above on a medium customarily used for software interchange; or,
140
141 b) Accompany it with a written offer, valid for at least three
142 years, to give any third party, for a charge no more than your
143 cost of physically performing source distribution, a complete
144 machine-readable copy of the corresponding source code, to be
145 distributed under the terms of Sections 1 and 2 above on a medium
146 customarily used for software interchange; or,
147
148 c) Accompany it with the information you received as to the offer
149 to distribute corresponding source code. (This alternative is
150 allowed only for noncommercial distribution and only if you
151 received the program in object code or executable form with such
152 an offer, in accord with Subsection b above.)
153
154 The source code for a work means the preferred form of the work for
155 making modifications to it. For an executable work, complete source
156 code means all the source code for all modules it contains, plus any
157 associated interface definition files, plus the scripts used to
158 control compilation and installation of the executable. However, as a
159 special exception, the source code distributed need not include
160 anything that is normally distributed (in either source or binary
161 form) with the major components (compiler, kernel, and so on) of the
162 operating system on which the executable runs, unless that component
163 itself accompanies the executable.
164
165 If distribution of executable or object code is made by offering
166 access to copy from a designated place, then offering equivalent
167 access to copy the source code from the same place counts as
168 distribution of the source code, even though third parties are not
169 compelled to copy the source along with the object code.
170
171 4. You may not copy, modify, sublicense, or distribute the Program
172 except as expressly provided under this License. Any attempt
173 otherwise to copy, modify, sublicense or distribute the Program is
174 void, and will automatically terminate your rights under this License.
175 However, parties who have received copies, or rights, from you under
176 this License will not have their licenses terminated so long as such
177 parties remain in full compliance.
178
179 5. You are not required to accept this License, since you have not
180 signed it. However, nothing else grants you permission to modify or
181 distribute the Program or its derivative works. These actions are
182 prohibited by law if you do not accept this License. Therefore, by
183 modifying or distributing the Program (or any work based on the
184 Program), you indicate your acceptance of this License to do so, and
185 all its terms and conditions for copying, distributing or modifying
186 the Program or works based on it.
187
188 6. Each time you redistribute the Program (or any work based on the
189 Program), the recipient automatically receives a license from the
190 original licensor to copy, distribute or modify the Program subject to
191 these terms and conditions. You may not impose any further
192 restrictions on the recipients' exercise of the rights granted herein.
193 You are not responsible for enforcing compliance by third parties to
194 this License.
195
196 7. If, as a consequence of a court judgment or allegation of patent
197 infringement or for any other reason (not limited to patent issues),
198 conditions are imposed on you (whether by court order, agreement or
199 otherwise) that contradict the conditions of this License, they do not
200 excuse you from the conditions of this License. If you cannot
201 distribute so as to satisfy simultaneously your obligations under this
202 License and any other pertinent obligations, then as a consequence you
203 may not distribute the Program at all. For example, if a patent
204 license would not permit royalty-free redistribution of the Program by
205 all those who receive copies directly or indirectly through you, then
206 the only way you could satisfy both it and this License would be to
207 refrain entirely from distribution of the Program.
208
209 If any portion of this section is held invalid or unenforceable under
210 any particular circumstance, the balance of the section is intended to
211 apply and the section as a whole is intended to apply in other
212 circumstances.
213
214 It is not the purpose of this section to induce you to infringe any
215 patents or other property right claims or to contest validity of any
216 such claims; this section has the sole purpose of protecting the
217 integrity of the free software distribution system, which is
218 implemented by public license practices. Many people have made
219 generous contributions to the wide range of software distributed
220 through that system in reliance on consistent application of that
221 system; it is up to the author/donor to decide if he or she is willing
222 to distribute software through any other system and a licensee cannot
223 impose that choice.
224
225 This section is intended to make thoroughly clear what is believed to
226 be a consequence of the rest of this License.
227
228 8. If the distribution and/or use of the Program is restricted in
229 certain countries either by patents or by copyrighted interfaces, the
230 original copyright holder who places the Program under this License
231 may add an explicit geographical distribution limitation excluding
232 those countries, so that distribution is permitted only in or among
233 countries not thus excluded. In such case, this License incorporates
234 the limitation as if written in the body of this License.
235
236 9. The Free Software Foundation may publish revised and/or new versions
237 of the General Public License from time to time. Such new versions will
238 be similar in spirit to the present version, but may differ in detail to
239 address new problems or concerns.
240
241 Each version is given a distinguishing version number. If the Program
242 specifies a version number of this License which applies to it and "any
243 later version", you have the option of following the terms and conditions
244 either of that version or of any later version published by the Free
245 Software Foundation. If the Program does not specify a version number of
246 this License, you may choose any version ever published by the Free Software
247 Foundation.
248
249 10. If you wish to incorporate parts of the Program into other free
250 programs whose distribution conditions are different, write to the author
251 to ask for permission. For software which is copyrighted by the Free
252 Software Foundation, write to the Free Software Foundation; we sometimes
253 make exceptions for this. Our decision will be guided by the two goals
254 of preserving the free status of all derivatives of our free software and
255 of promoting the sharing and reuse of software generally.
256
257 NO WARRANTY
258
259 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
260 FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
261 OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
262 PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
263 OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
264 MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
265 TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
266 PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
267 REPAIR OR CORRECTION.
268
269 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
270 WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
271 REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
272 INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
273 OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
274 TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
275 YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
276 PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
277 POSSIBILITY OF SUCH DAMAGES.
278
279 END OF TERMS AND CONDITIONS
280
281 How to Apply These Terms to Your New Programs
282
283 If you develop a new program, and you want it to be of the greatest
284 possible use to the public, the best way to achieve this is to make it
285 free software which everyone can redistribute and change under these terms.
286
287 To do so, attach the following notices to the program. It is safest
288 to attach them to the start of each source file to most effectively
289 convey the exclusion of warranty; and each file should have at least
290 the "copyright" line and a pointer to where the full notice is found.
291
292 {description}
293 Copyright (C) {year} {fullname}
294
295 This program is free software; you can redistribute it and/or modify
296 it under the terms of the GNU General Public License as published by
297 the Free Software Foundation; either version 2 of the License, or
298 (at your option) any later version.
299
300 This program is distributed in the hope that it will be useful,
301 but WITHOUT ANY WARRANTY; without even the implied warranty of
302 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
303 GNU General Public License for more details.
304
305 You should have received a copy of the GNU General Public License along
306 with this program; if not, write to the Free Software Foundation, Inc.,
307 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
308
309 Also add information on how to contact you by electronic and paper mail.
310
311 If the program is interactive, make it output a short notice like this
312 when it starts in an interactive mode:
313
314 Gnomovision version 69, Copyright (C) year name of author
315 Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
316 This is free software, and you are welcome to redistribute it
317 under certain conditions; type `show c' for details.
318
319 The hypothetical commands `show w' and `show c' should show the appropriate
320 parts of the General Public License. Of course, the commands you use may
321 be called something other than `show w' and `show c'; they could even be
322 mouse-clicks or menu items--whatever suits your program.
323
324 You should also get your employer (if you work as a programmer) or your
325 school, if any, to sign a "copyright disclaimer" for the program, if
326 necessary. Here is a sample; alter the names:
327
328 Yoyodyne, Inc., hereby disclaims all copyright interest in the program
329 `Gnomovision' (which makes passes at compilers) written by James Hacker.
330
331 {signature of Ty Coon}, 1 April 1989
332 Ty Coon, President of Vice
333
334 This General Public License does not permit incorporating your program into
335 proprietary programs. If your program is a subroutine library, you may
336 consider it more useful to permit linking proprietary applications with the
337 library. If this is what you want to do, use the GNU Lesser General
338 Public License instead of this License.
0 GIT_DATE := $(shell git log -1 --date=short --pretty='%cd' | tr -d -)
1 GIT_HASH := $(shell git rev-parse HEAD)
2
3 BUILD_FLAGS := -ldflags "-X main.gitDate=$(GIT_DATE) -X main.gitHash=$(GIT_HASH)"
4
5 PLATFORMS := linux/amd64 linux/386 linux/arm darwin/amd64 windows/amd64 windows/386 openbsd/amd64
6 SOURCES := $(shell find . -maxdepth 1 -type f -name "*.go")
7 ALL_SOURCES = $(shell find . -type f -name '*.go')
8
9 temp = $(subst /, ,$@)
10 os = $(word 1, $(temp))
11 arch = $(word 2, $(temp))
12 ext = $(shell if [ "$(os)" = "windows" ]; then echo ".exe"; fi)
13
14 .PHONY: all release fmt clean serv $(PLATFORMS)
15
16 all: certgraph
17
18 release: $(PLATFORMS)
19 rm -r build/bin/
20
21 certgraph: $(SOURCES) $(ALL_SOURCES)
22 go build $(BUILD_FLAGS) -o $@ $(SOURCES)
23
24 $(PLATFORMS): $(SOURCES)
25 CGO_ENABLED=0 GOOS=$(os) GOARCH=$(arch) go build $(BUILD_FLAGS) -o 'build/bin/$(os)/$(arch)/certgraph$(ext)' $(SOURCES)
26 mkdir -p build/$(GIT_DATE)/; cd build/bin/$(os)/$(arch)/; zip -r ../../../$(GIT_DATE)/certgraph-$(os)-$(arch)-$(GIT_DATE).zip .; cd ../../../
27
28 fmt:
29 gofmt -s -w -l .
30
31 install: $(SOURCES) $(ALL_SOURCES)
32 go install $(BUILD_FLAGS)
33
34 clean:
35 rm -r certgraph build/
36
37 serv:
38 (cd docs; python -m SimpleHTTPServer)
0 # CertGraph
1 ### A tool to crawl the graph of certificate Alternate Names
2
3 CertGraph crawls SSL certificates creating a directed graph where each domain is a node and the certificate alternative names for that domain's certificate are the edges to other domain nodes. New domains are printed as they are found. In Detailed mode upon completion the Graph's adjacency list is printed.
4
5 Crawling defaults to collecting certificate by connecting over TCP, however there are multiple drivers that can search [Certificate Transparency](https://www.certificate-transparency.org/) logs.
6
7 This tool was designed to be used for host name enumeration via SSL certificates, but it can also show you a "chain" of trust between domains and the certificates that re-used between them.
8
9 [Blog post with more information](https://lanrat.com/certgraph/)
10
11 ## Usage
12 ```
13 Usage of ./certgraph: [OPTION]... HOST...
14 https://github.com/lanrat/certgraph
15 OPTIONS:
16 -cdn
17 include certificates from CDNs
18 -ct-expired
19 include expired certificates in certificate transparency search
20 -ct-subdomains
21 include sub-domains in certificate transparency search
22 -depth uint
23 maximum BFS depth to go (default 5)
24 -details
25 print details about the domains crawled
26 -driver string
27 driver to use [crtsh, google, http, smtp] (default "http")
28 -json
29 print the graph as json, can be used for graph in web UI
30 -ns
31 check for NS records to determine if domain is registered
32 -parallel uint
33 number of certificates to retrieve in parallel (default 10)
34 -sanscap int
35 maximum number of uniq TLD+1 domains in certificate to include, 0 has no limit (default 80)
36 -save string
37 save certs to folder in PEM format
38 -timeout uint
39 tcp timeout in seconds (default 10)
40 -tldplus1
41 for every domain found, add tldPlus1 of the domain's parent
42 -verbose
43 verbose logging
44 -version
45 print version and exit
46 ```
47
48 ## Drivers
49
50 CertGraph has multiple options for querying SSL certificates. The driver is responsible for retrieving the certificates for a given domain. Currently there are the following drivers:
51
52 * **http** this is the default driver which works by connecting to the hosts over HTTPS and retrieving the certificates from the SSL connection
53
54 * **smtp** like the *http* driver, but connects over port 25 and issues the *starttls* command to retrieve the certificates from the SSL connection
55
56 * **crtsh** this driver searches Certificate Transparency logs via [crt.sh](https://crt.sh/). No packets are sent to any of the domains when using this driver
57
58 * **google** this is another Certificate Transparency driver that behaves like *crtsh* but uses the [Google Certificate Transparency Lookup Tool](https://transparencyreport.google.com/https/certificates)
59
60
61 ## Example
62 ```
63 $ ./certgraph -details eff.org
64 eff.org 0 Good 42E3E4605D8BB4608EB64936E2176A98B97EBF2E0F8F93A64A6640713C7D4325
65 maps.eff.org 1 Good 42E3E4605D8BB4608EB64936E2176A98B97EBF2E0F8F93A64A6640713C7D4325
66 https-everywhere-atlas.eff.org 1 Good 42E3E4605D8BB4608EB64936E2176A98B97EBF2E0F8F93A64A6640713C7D4325
67 httpse-atlas.eff.org 1 Good 42E3E4605D8BB4608EB64936E2176A98B97EBF2E0F8F93A64A6640713C7D4325
68 atlas.eff.org 1 Good 42E3E4605D8BB4608EB64936E2176A98B97EBF2E0F8F93A64A6640713C7D4325
69 kittens.eff.org 1 Good 42E3E4605D8BB4608EB64936E2176A98B97EBF2E0F8F93A64A6640713C7D4325
70 ```
71 The above output represents the adjacency list for the graph for the root domain `eff.org`. The adjacency list is in the form:
72 `Node Depth Status Cert-Fingerprint`
73
74 ## [Releases](https://github.com/lanrat/certgraph/releases)
75
76 Precompiled releases will occasionally be uploaded to the [releases github page](https://github.com/lanrat/certgraph/releases). https://github.com/lanrat/certgraph/releases
77
78 Also available in [BlackArch](https://blackarch.org).
79
80 ## Compiling
81
82 To compile certgraph you must have a working go 1.11 or newer compiler on your system, as certgraph makes use of go's modules for dependencies.
83 To compile for the running system compilation is as easy as running make
84 ```
85 certgraph$ make
86 go build -o certgraph certgraph.go
87 ```
88
89 Alternatively you can use `go get` to install with this one-liner:
90 ```
91 go get -u github.com/lanrat/certgraph
92 ```
93
94 ## [Web UI](https://lanrat.github.io/certgraph/)
95
96 A web UI is provided in the docs folder and is accessible at the github pages url [https://lanrat.github.io/certgraph/](https://lanrat.github.io/certgraph/).
97
98 The web UI takes the output provided with the `-json` flag.
99 The JSON graph can be sent to the web interface as an uploaded file, remote URL, or as the query string using the data variable.
100
101 ### [Example 1: eff.org](https://lanrat.github.io/certgraph/?data=https://gist.githubusercontent.com/lanrat/8187d01793bf3e578d76495182654206/raw/c49741b5206d81935febdf563452cc4346381e52/eff.json)
102
103 [![eff.org graph](https://cloud.githubusercontent.com/assets/164192/20861413/6ba0fcca-b944-11e6-857f-ddd613130ea3.png)](https://lanrat.github.io/certgraph/?data=https://gist.githubusercontent.com/lanrat/8187d01793bf3e578d76495182654206/raw/c49741b5206d81935febdf563452cc4346381e52/eff.json)
104
105 ### [Example 2: google.com](https://lanrat.github.io/certgraph/?data=https://gist.githubusercontent.com/lanrat/1ab1e78aaf5798049650d8d8ad7b58a1/raw/426d3a2498626014cb5ba2856ad0899787e4103f/google.json)
106
107 [![google.com graph](https://cloud.githubusercontent.com/assets/164192/19752837/16cb8302-9bb5-11e6-810d-ea34594a63ef.png)](https://lanrat.github.io/certgraph/?data=https://gist.githubusercontent.com/lanrat/1ab1e78aaf5798049650d8d8ad7b58a1/raw/426d3a2498626014cb5ba2856ad0899787e4103f/google.json)
108
109 ### [Example 3: whitehouse.gov](https://lanrat.github.io/certgraph/?data=https://gist.githubusercontent.com/lanrat/96c47dfee0faaaad633cc830b7e3b997/raw/3c79fed837cb3202e220de21d2a8eb128f4bbd9f/whitehouse.json)
110
111 [![whitehouse.gov graph](https://cloud.githubusercontent.com/assets/164192/20861407/4775ff26-b944-11e6-888c-4d93e3333494.png)](https://lanrat.github.io/certgraph/?data=https://gist.githubusercontent.com/lanrat/96c47dfee0faaaad633cc830b7e3b997/raw/3c79fed837cb3202e220de21d2a8eb128f4bbd9f/whitehouse.json)
112
0 package main
1
2 import (
3 "encoding/json"
4 "flag"
5 "fmt"
6 "net/url"
7 "os"
8 "strings"
9 "sync"
10 "time"
11
12 "github.com/lanrat/certgraph/dns"
13 "github.com/lanrat/certgraph/driver"
14 "github.com/lanrat/certgraph/driver/crtsh"
15 "github.com/lanrat/certgraph/driver/google"
16 "github.com/lanrat/certgraph/driver/http"
17 "github.com/lanrat/certgraph/driver/smtp"
18 "github.com/lanrat/certgraph/graph"
19 )
20
21 var (
22 gitDate = "none"
23 gitHash = "master"
24 certGraph = graph.NewCertGraph()
25 )
26
27 var certDriver driver.Driver
28
29 // config & flags
30 var config struct {
31 timeout time.Duration
32 verbose bool
33 maxDepth uint
34 parallel uint
35 savePath string
36 details bool
37 printJSON bool
38 driver string
39 includeCTSubdomains bool
40 includeCTExpired bool
41 cdn bool
42 maxSANsSize int
43 tldPlus1 bool
44 updatePSL bool
45 checkDNS bool
46 printVersion bool
47 }
48
49 func init() {
50 var timeoutSeconds uint
51 flag.BoolVar(&config.printVersion, "version", false, "print version and exit")
52 flag.UintVar(&timeoutSeconds, "timeout", 10, "tcp timeout in seconds")
53 flag.BoolVar(&config.verbose, "verbose", false, "verbose logging")
54 flag.StringVar(&config.driver, "driver", "http", fmt.Sprintf("driver to use [%s]", strings.Join(driver.Drivers, ", ")))
55 flag.BoolVar(&config.includeCTSubdomains, "ct-subdomains", false, "include sub-domains in certificate transparency search")
56 flag.BoolVar(&config.includeCTExpired, "ct-expired", false, "include expired certificates in certificate transparency search")
57 flag.IntVar(&config.maxSANsSize, "sanscap", 80, "maximum number of uniq TLD+1 domains in certificate to include, 0 has no limit")
58 flag.BoolVar(&config.cdn, "cdn", false, "include certificates from CDNs")
59 flag.BoolVar(&config.checkDNS, "dns", false, "check for DNS records to determine if domain is registered")
60 flag.BoolVar(&config.tldPlus1, "tldplus1", false, "for every domain found, add tldPlus1 of the domain's parent")
61 flag.BoolVar(&config.updatePSL, "updatepsl", false, "Update the default Public Suffix List")
62 flag.UintVar(&config.maxDepth, "depth", 5, "maximum BFS depth to go")
63 flag.UintVar(&config.parallel, "parallel", 10, "number of certificates to retrieve in parallel")
64 flag.BoolVar(&config.details, "details", false, "print details about the domains crawled")
65 flag.BoolVar(&config.printJSON, "json", false, "print the graph as json, can be used for graph in web UI")
66 flag.StringVar(&config.savePath, "save", "", "save certs to folder in PEM format")
67 flag.Usage = func() {
68 fmt.Fprintf(os.Stderr, "Usage of %s: [OPTION]... HOST...\n\thttps://github.com/lanrat/certgraph\nOPTIONS:\n", os.Args[0])
69 flag.PrintDefaults()
70 }
71 flag.Parse()
72 config.timeout = time.Duration(timeoutSeconds) * time.Second
73 }
74
75 func main() {
76 // check for version flag
77 if config.printVersion {
78 fmt.Println(version())
79 return
80 }
81
82 // print usage if no domain passed
83 if flag.NArg() < 1 {
84 flag.Usage()
85 return
86 }
87
88 // cant run on 0 threads
89 if config.parallel < 1 {
90 fmt.Fprintln(os.Stderr, "Must enter a positive number of parallel threads")
91 flag.Usage()
92 return
93 }
94
95 // update the public suffix list if required
96 if config.updatePSL {
97 err := dns.UpdatePublicSuffixList(config.timeout)
98 if err != nil {
99 e(err)
100 return
101 }
102 }
103
104 // add domains passed to startDomains
105 startDomains := make([]string, 0, 1)
106 for _, domain := range flag.Args() {
107 d := strings.ToLower(domain)
108 if len(d) > 0 {
109 startDomains = append(startDomains, cleanInput(d))
110 if config.tldPlus1 {
111 tldPlus1, err := dns.TLDPlus1(domain)
112 if err != nil {
113 continue
114 }
115 startDomains = append(startDomains, tldPlus1)
116 }
117 }
118 }
119
120 // set driver
121 err := setDriver(config.driver)
122 if err != nil {
123 fmt.Fprintln(os.Stderr, err)
124 return
125 }
126
127 // create the output directory if it does not exist
128 if len(config.savePath) > 0 {
129 err := os.MkdirAll(config.savePath, 0777)
130 if err != nil {
131 fmt.Fprintln(os.Stderr, err)
132 return
133 }
134 }
135
136 // perform breath-first-search on the graph
137 breathFirstSearch(startDomains)
138
139 // print the json output
140 if config.printJSON {
141 printJSONGraph()
142 }
143
144 v("Found", certGraph.NumDomains(), "domains")
145 v("Graph Depth:", certGraph.DomainDepth())
146 }
147
148 // setDriver sets the driver variable for the provided driver string and does any necessary driver prep work
149 // TODO make config generic and move this to driver module
150 func setDriver(driver string) error {
151 var err error
152 switch driver {
153 case "google":
154 certDriver, err = google.Driver(50, config.savePath, config.includeCTSubdomains, config.includeCTExpired)
155 case "crtsh":
156 certDriver, err = crtsh.Driver(1000, config.timeout, config.savePath, config.includeCTSubdomains, config.includeCTExpired)
157 case "http":
158 certDriver, err = http.Driver(config.timeout, config.savePath)
159 case "smtp":
160 certDriver, err = smtp.Driver(config.timeout, config.savePath)
161 default:
162 return fmt.Errorf("Unknown driver name: %s", config.driver)
163 }
164 return err
165 }
166
167 // verbose logging
168 func v(a ...interface{}) {
169 if config.verbose {
170 e(a...)
171 }
172 }
173
174 func e(a ...interface{}) {
175 if a != nil {
176 fmt.Fprintln(os.Stderr, a...)
177 }
178 }
179
180 // prints the graph as a json object
181 func printJSONGraph() {
182 jsonGraph := certGraph.GenerateMap()
183 jsonGraph["certgraph"] = generateGraphMetadata()
184
185 j, err := json.MarshalIndent(jsonGraph, "", "\t")
186 if err != nil {
187 fmt.Println(err)
188 return
189 }
190 fmt.Println(string(j))
191 }
192
193 // breathFirstSearch perform Breadth first search to build the graph
194 func breathFirstSearch(roots []string) {
195 var wg sync.WaitGroup
196 domainNodeInputChan := make(chan *graph.DomainNode, 5) // input queue
197 domainNodeOutputChan := make(chan *graph.DomainNode, 5) // output queue
198
199 // thread limit code
200 threadPass := make(chan bool, config.parallel)
201 for i := uint(0); i < config.parallel; i++ {
202 threadPass <- true
203 }
204
205 // thread to put root nodes/domains into queue
206 wg.Add(1)
207 go func() {
208 // the waitGroup Add and Done for this thread ensures that we don't exit before any of the inputs domains are put into the Queue
209 defer wg.Done()
210 for _, root := range roots {
211 wg.Add(1)
212 n := graph.NewDomainNode(root, 0)
213 n.Root = true
214 domainNodeInputChan <- n
215 }
216 }()
217 // thread to start all other threads from DomainChan
218 go func() {
219 for {
220 domainNode := <-domainNodeInputChan
221
222 // depth check
223 if domainNode.Depth > config.maxDepth {
224 v("Max depth reached, skipping:", domainNode.Domain)
225 wg.Done()
226 continue
227 }
228 // use certGraph.domains map as list of
229 // domains that are queued to be visited, or already have been
230
231 if _, found := certGraph.GetDomain(domainNode.Domain); !found {
232 certGraph.AddDomain(domainNode)
233 go func(domainNode *graph.DomainNode) {
234 defer wg.Done()
235 // wait for pass
236 <-threadPass
237 defer func() { threadPass <- true }()
238
239 // operate on the node
240 v("Visiting", domainNode.Depth, domainNode.Domain)
241 visit(domainNode)
242 domainNodeOutputChan <- domainNode
243 for _, neighbor := range certGraph.GetDomainNeighbors(domainNode.Domain, config.cdn, config.maxSANsSize) {
244 wg.Add(1)
245 domainNodeInputChan <- graph.NewDomainNode(neighbor, domainNode.Depth+1)
246 if config.tldPlus1 {
247 tldPlus1, err := dns.TLDPlus1(neighbor)
248 if err != nil {
249 continue
250 }
251 wg.Add(1)
252 domainNodeInputChan <- graph.NewDomainNode(tldPlus1, domainNode.Depth+1)
253 }
254 }
255 }(domainNode)
256 } else {
257 wg.Done()
258 }
259 }
260 }()
261
262 // save/output thread
263 done := make(chan bool)
264 go func() {
265 for {
266 domainNode, more := <-domainNodeOutputChan
267 if more {
268 if !config.printJSON {
269 printNode(domainNode)
270 } else if config.details {
271 fmt.Fprintln(os.Stderr, domainNode)
272 }
273 } else {
274 done <- true
275 return
276 }
277 }
278 }()
279
280 wg.Wait() // wait for querying to finish
281 close(domainNodeOutputChan)
282 <-done // wait for save to finish
283 }
284
285 // visit visits each node and get and set its neighbors
286 func visit(domainNode *graph.DomainNode) {
287 // check NS if necessary
288 if config.checkDNS {
289 _, err := domainNode.CheckForDNS(config.timeout)
290 if err != nil {
291 v("CheckForNS", err)
292 }
293 }
294
295 // perform cert search
296 // TODO do pagination in multiple threads to not block on long searches
297 results, err := certDriver.QueryDomain(domainNode.Domain)
298 if err != nil {
299 // this is VERY common to error, usually this is a DNS or tcp connection related issue
300 // we will skip the domain if we can't query it
301 v("QueryDomain", domainNode.Domain, err)
302 return
303 }
304 statuses := results.GetStatus()
305 domainNode.AddStatusMap(statuses)
306 relatedDomains, err := results.GetRelated()
307 if err != nil {
308 v("GetRelated", domainNode.Domain, err)
309 return
310 }
311 domainNode.AddRelatedDomains(relatedDomains)
312
313 // TODO parallelize this
314 // TODO fix printing domains as they are found with new driver
315 // add cert nodes to graph
316 fingerprintMap, err := results.GetFingerprints()
317 if err != nil {
318 v("GetFingerprints", err)
319 return
320 }
321
322 // fingerprints for the domain queried
323 fingerprints := fingerprintMap[domainNode.Domain]
324 for _, fp := range fingerprints {
325 // add certnode to graph
326 certNode, exists := certGraph.GetCert(fp)
327 if !exists {
328 // get cert details
329 certResult, err := results.QueryCert(fp)
330 if err != nil {
331 v("QueryCert", err)
332 continue
333 }
334
335 certNode = certNodeFromCertResult(certResult)
336 certGraph.AddCert(certNode)
337 }
338
339 certNode.AddFound(certDriver.GetName())
340 domainNode.AddCertFingerprint(certNode.Fingerprint, certDriver.GetName())
341 }
342
343 // we dont process any other certificates returned, they will be collected
344 // when we process the related domains
345 }
346
347 func printNode(domainNode *graph.DomainNode) {
348 if config.details {
349 fmt.Fprintln(os.Stdout, domainNode)
350 } else {
351 fmt.Fprintln(os.Stdout, domainNode.Domain)
352 }
353 if config.checkDNS && !domainNode.HasDNS {
354 // TODO print this in a better way
355 // TODO for debugging
356 realDomain, _ := dns.TLDPlus1(domainNode.Domain)
357 fmt.Fprintf(os.Stdout, "* Missing DNS for: %s\n", realDomain)
358
359 }
360 }
361
362 // certNodeFromCertResult convert certResult to certNode
363 func certNodeFromCertResult(certResult *driver.CertResult) *graph.CertNode {
364 certNode := &graph.CertNode{
365 Fingerprint: certResult.Fingerprint,
366 Domains: certResult.Domains,
367 }
368 return certNode
369 }
370
371 // generates metadata for the JSON output
372 // TODO map all config json
373 func generateGraphMetadata() map[string]interface{} {
374 data := make(map[string]interface{})
375 data["version"] = version()
376 data["website"] = "https://lanrat.github.io/certgraph/"
377 data["scan_date"] = time.Now().UTC()
378 data["command"] = strings.Join(os.Args, " ")
379 options := make(map[string]interface{})
380 options["parallel"] = config.parallel
381 options["driver"] = config.driver
382 options["ct_subdomains"] = config.includeCTSubdomains
383 options["ct_expired"] = config.includeCTExpired
384 options["sanscap"] = config.maxSANsSize
385 options["cdn"] = config.cdn
386 options["timeout"] = config.timeout
387 data["options"] = options
388 return data
389 }
390
391 // returns the version string
392 func version() string {
393 return fmt.Sprintf("Git commit: %s [%s]", gitDate, gitHash)
394 }
395
396 // cleanInput attempts to parse the input string as a url to extract the hostname
397 // if it fails, then the input string is returned
398 // also removes tailing '.'
399 func cleanInput(host string) string {
400 host = strings.TrimSuffix(host, ".")
401 u, err := url.Parse(host)
402 if err != nil {
403 return host
404 }
405 hostname := u.Hostname()
406 if hostname == "" {
407 return host
408 }
409 return hostname
410 }
0 package dns
1
2 import (
3 "context"
4 "net"
5 "time"
6 )
7
8 var (
9 dnsCache = make(map[string]bool)
10 dnsResolver = &net.Resolver{}
11 )
12
13 func init() {
14 //dnsResolver.PreferGo = true
15 dnsResolver.StrictErrors = false
16 }
17
18 func noSuchHostDNSError(err error) bool {
19 dnsErr, ok := err.(*net.DNSError)
20 if !ok {
21 // not a DNSError
22 return false
23 }
24 return dnsErr.Err == "no such host"
25 }
26
27 // HasRecords does NS, CNAME, A, and AAAA lookups with a timeout
28 // returns error when no NS found, does not use TLDPlus1
29 func HasRecords(domain string, timeout time.Duration) (bool, error) {
30 ctx, cancel := context.WithTimeout(context.Background(), timeout)
31 defer cancel()
32
33 // first check for NS
34 ns, err := dnsResolver.LookupNS(ctx, domain)
35 if err != nil && !noSuchHostDNSError(err) {
36 //fmt.Println("NS error ", err)
37 return false, err
38 }
39 if len(ns) > 0 {
40 //fmt.Printf("Found %d NS for %s\n", len(ns), domain)
41 return true, nil
42 }
43
44 // next check for CNAME
45 cname, err := dnsResolver.LookupCNAME(ctx, domain)
46 if err != nil && !noSuchHostDNSError(err) {
47 //fmt.Println("cname error ", err)
48 return false, err
49 }
50 if len(cname) > 2 {
51 //fmt.Printf("found CNAME %s for %s\n", cname, domain)
52 return true, nil
53 }
54
55 // next check for IP
56 addrs, err := dnsResolver.LookupHost(ctx, domain)
57 if err != nil && !noSuchHostDNSError(err) {
58 //fmt.Println("ip error ", err)
59 return false, err
60 }
61 if len(addrs) > 0 {
62 //fmt.Printf("Found %d IPs for %s\n", len(addrs), domain)
63 return true, nil
64 }
65
66 //fmt.Printf("Found no DNS records for %s\n", domain)
67 return false, nil
68 }
69
70 // HasRecordsCache returns true if the domain has no DNS records (at the tldplus1 level)
71 // uses a cache to store results to prevent lots of DNS lookups
72 func HasRecordsCache(domain string, timeout time.Duration) (bool, error) {
73 domain, err := TLDPlus1(domain)
74 if err != nil {
75 return false, err
76 }
77 hasDNS, found := dnsCache[domain]
78 if found {
79 return hasDNS, nil
80 }
81 hasRecords, err := HasRecords(domain, timeout)
82 if err != nil {
83 dnsCache[domain] = hasRecords
84 }
85 return hasRecords, err
86 }
0 package dns
1
2 import (
3 "net/http"
4 "time"
5
6 "github.com/weppos/publicsuffix-go/publicsuffix"
7 )
8
9 var (
10 suffixListFindOptions = &publicsuffix.FindOptions{
11 IgnorePrivate: true,
12 DefaultRule: publicsuffix.DefaultRule,
13 }
14 suffixListURL = "https://publicsuffix.org/list/public_suffix_list.dat"
15 suffixList = publicsuffix.DefaultList
16 nsCache = make(map[string]bool)
17 )
18
19 // UpdatePublicSuffixList gets a new copy of the public suffix list from the internat and updates the built in copy with the new rules
20 func UpdatePublicSuffixList(timeout time.Duration) error {
21 suffixListParseOptions := &publicsuffix.ParserOption{
22 PrivateDomains: !suffixListFindOptions.IgnorePrivate,
23 }
24 client := http.Client{
25 Timeout: timeout,
26 }
27 resp, err := client.Get(suffixListURL)
28 if err != nil {
29 return err
30 }
31 defer resp.Body.Close()
32 newSuffixList := publicsuffix.NewList()
33 newSuffixList.Load(resp.Body, suffixListParseOptions)
34 suffixList = newSuffixList
35 return err
36 }
37
38 // TLDPlus1 returns TLD+1 of domain
39 func TLDPlus1(domain string) (string, error) {
40 return publicsuffix.DomainFromListWithOptions(suffixList, domain, suffixListFindOptions)
41 }
0 <!DOCTYPE html>
1 <html>
2 <head>
3 <meta charset="utf-8">
4 <meta http-equiv="X-UA-Compatible" content="IE=edge">
5 <meta name="viewport" content="width=device-width, initial-scale=1">
6 <title>CertGraph</title>
7 <link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/cerulean/bootstrap.min.css" rel="stylesheet" integrity="sha384-zF4BRsG/fLiTGfR9QL82DrilZxrwgY/+du4p/c7J72zZj+FLYq4zY00RylP9ZjiT" crossorigin="anonymous">
8 <script src="https://d3js.org/d3.v4.min.js"></script>
9 <style type="text/css">
10 .links line {
11 stroke-opacity: 0.6;
12 stroke-width: 1px;
13 fill: none;
14 }
15 .nodes circle {
16 stroke: #333;
17 stroke-width: 1.5px;
18 }
19
20 .upload-drop-zone {
21 height: 200px;
22 border-width: 2px;
23 margin-bottom: 20px;
24 color: #ccc;
25 border-style: dashed;
26 border-color: #ccc;
27 line-height: 200px;
28 text-align: center
29 }
30 .upload-drop-zone.drop {
31 color: #222;
32 border-color: #222;
33 }
34 </style>
35 </head>
36 <body>
37 <div class="container">
38
39 <nav class="navbar navbar-inverse">
40 <div class="container-fluid">
41 <div class="navbar-header">
42 <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
43 <span class="sr-only">Toggle navigation</span>
44 <span class="icon-bar"></span>
45 <span class="icon-bar"></span>
46 <span class="icon-bar"></span>
47 </button>
48 <a class="navbar-brand" href="#">CertGraph</a>
49 </div>
50 <div id="navbar" class="navbar-collapse collapse">
51 <ul class="nav navbar-nav">
52 <!-- <li><a href="#">Graph</a></li> -->
53 </ul>
54 <ul class="nav navbar-nav navbar-right">
55 <li class="dropdown">
56 <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Data <span class="caret"></span></a>
57 <ul class="dropdown-menu" role="menu">
58 <li><a href="#" data-toggle="modal" data-target="#URLmodal">URL</a></li>
59 <li><a href="#" data-toggle="modal" data-target="#Pastemodal">Paste</a></li>
60 <li><a href="#" data-toggle="modal" data-target="#Filemodal">File Upload</a></li>
61 </ul>
62 </li>
63 </ul>
64 </div>
65 </div>
66 </nav>
67
68 <div class="panel panel-info">
69 <div class="panel-heading">
70 <h3 class="panel-title pull-left">Graph</h3>
71 <div class="pull-right"><a href="#" class="btn btn-primary btn-xs" id="generate">Download SVG</a></div>
72 <div class="clearfix"></div>
73 </div>
74 <svg id="graph" width="100%" height="500"></svg>
75 </div>
76
77 <div class="panel panel-info">
78 <div class="panel-heading">Info</div>
79 <div class="panel-body" id="node-info">
80 </div>
81 </div>
82
83 <ul class="nav nav-tabs">
84 <li class="active"><a href="#domains" data-toggle="tab" aria-expanded="false">Domains</a></li>
85 <li class=""><a href="#certificates" data-toggle="tab" aria-expanded="true">Certificates</a></li>
86 </ul>
87 <div id="myTabContent" class="tab-content panel-body">
88 <div class="tab-pane fade active in" id="domains">
89
90 <table class="table table-striped table-hover ">
91 <thead>
92 <tr>
93 <th>#</th>
94 <th>Domain</th>
95 <th>Status</th>
96 <th>Lookup</th>
97 </tr>
98 </thead>
99 <tbody id="domain-list">
100 </tbody>
101 </table>
102
103 </div>
104 <div class="tab-pane fade" id="certificates">
105
106 <table class="table table-striped table-hover ">
107 <thead>
108 <tr>
109 <th>#</th>
110 <th>Hash</th>
111 <th>Lookup</th>
112 </tr>
113 </thead>
114 <tbody id="cert-list">
115 </tbody>
116 </table>
117
118 </div>
119
120 </div>
121
122 <footer>
123 <hr>
124 <div class="row">
125 <div class="col-xs-10"><a href="https://github.com/lanrat/certgraph">CertGraph</a></div>
126 </div>
127 </footer>
128
129 <!-- URL Modal -->
130 <div class="modal fade" id="URLmodal" role="dialog">
131 <div class="modal-dialog">
132 <!-- Modal content-->
133 <div class="modal-content">
134 <div class="modal-header">
135 <button type="button" class="close" data-dismiss="modal">&times;</button>
136 <h4 class="modal-title">Data URL</h4>
137 </div>
138 <div class="modal-body">
139 <label for="inputURL" class="col-lg-2 control-label">URL</label>
140 <div class="col-lg-10">
141 <input type="text" class="form-control" id="inputURL" placeholder="https://domain.com/data.json">
142 </div>
143 </div>
144 <div class="modal-footer">
145 <button type="button" class="btn btn-primary" data-dismiss="modal" id="loadURL">Load</button>
146 <button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
147 </div>
148 </div>
149 </div>
150 </div> <!-- /URL Modal -->
151
152 <!-- Paste Modal -->
153 <div class="modal fade" id="Pastemodal" role="dialog">
154 <div class="modal-dialog">
155 <!-- Modal content-->
156 <div class="modal-content">
157 <div class="modal-header">
158 <button type="button" class="close" data-dismiss="modal">&times;</button>
159 <h4 class="modal-title">JSON Data</h4>
160 </div>
161 <div class="modal-body">
162 <label for="inputPaste" class="col-lg-2 control-label">URL</label>
163 <div class="col-lg-10">
164 <textarea class="form-control" rows="10" id="inputPaste"></textarea>
165 </div>
166 </div>
167 <div class="modal-footer">
168 <button type="button" class="btn btn-primary" data-dismiss="modal" id="loadPaste">Load</button>
169 <button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
170 </div>
171 </div>
172 </div>
173 </div> <!-- /Paste Modal -->
174
175 <!-- File Modal -->
176 <div class="modal fade" id="Filemodal" role="dialog">
177 <div class="modal-dialog">
178 <!-- Modal content-->
179 <div class="modal-content">
180 <div class="modal-header">
181 <button type="button" class="close" data-dismiss="modal">&times;</button>
182 <h4 class="modal-title">JSON File</h4>
183 </div>
184 <div class="modal-body">
185 <label for="inputFile" class="col-lg-2 control-label">File</label>
186 <div class="col-lg-10">
187 <input type="file" class="form-control file" id="inputFile">
188 </div>
189 <br/>
190 <label for="drop-zone" class="col-lg-2 control-label">Or drag and drop a file below</label>
191 <div class="upload-drop-zone" id="drop-zone">
192 Just drag and drop a JSON file here
193 </div>
194 </div>
195 <div class="modal-footer">
196 <button type="button" class="btn btn-primary" data-dismiss="modal" id="loadFile">Load</button>
197 <button type="button" class="btn btn-default" data-dismiss="modal" id="fileClose">Close</button>
198 </div>
199 </div>
200 </div>
201 </div> <!-- /File Modal -->
202
203 </div> <!-- /container-->
204 <script src="//code.jquery.com/jquery-1.11.3.min.js"></script>
205 <script src="//maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
206 <script src="//cdn.rawgit.com/eligrey/FileSaver.js/e9d941381475b5df8b7d7691013401e171014e89/FileSaver.min.js"></script>
207 <script>
208 var svg = d3.select("svg");
209 var width = window.innerWidth-100;
210 //var width = svg.attr("width");
211 var height = svg.attr("height");
212
213 /*var svgElem = document.getElementById("graph");
214 var width = svgElem.width.animVal.value,
215 height = svgElem.height.animVal.value;
216 console.log(width, height);*/
217
218 // TODO THIS
219 // http://www.coppelia.io/2014/07/an-a-to-z-of-extra-features-for-the-d3-force-layout/
220
221 var color = d3.scaleOrdinal(d3.schemeCategory10);
222 var simulation;
223
224 svg = svg.call(d3.zoom().on("zoom", zoomed)).append("g");
225
226 svg.append("defs").append("marker")
227 .attr("id", "arrow")
228 .attr("viewBox", "0 -5 10 10")
229 .attr("refX", 20)
230 .attr("refY", 0)
231 .attr("markerWidth", 8)
232 .attr("markerHeight", 8)
233 .attr("orient", "auto")
234 //.attr("stroke", function(d) { return color(d.type); })
235 .append("svg:path")
236 .attr("d", "M0,-5L10,0L0,5");
237
238 function resetGraph() {
239 d3.select("g").selectAll("*").remove();
240 createTables();
241
242 // reset info
243 var el = document.getElementById("node-info");
244 el.innerText = "Click on a node in the graph to view details.";
245
246 // redo layout
247 simulation = d3.forceSimulation()
248 .force("link", d3.forceLink().id(function(d) { return d.id; }))
249 .force("charge", d3.forceManyBody().strength(-100))
250 .force("center", d3.forceCenter(width / 2, height / 2));
251 }
252
253 function createGraph (error, graph) {
254 if (error) throw error;
255
256 var link = svg.append("g")
257 .attr("class", "links")
258 .selectAll("line")
259 .data(graph.links)
260 .enter().append("line")
261 .attr("stroke", function(d) { return color(d.type); })
262 .attr("marker-end", "url(#arrow)");
263
264 var text = svg.append("g").attr("class", "labels").selectAll("g")
265 .data(graph.nodes)
266 .enter().append("g");
267
268 text.append("text")
269 .attr("x", 14)
270 .attr("y", ".31em")
271 .style("font-family", "sans-serif")
272 .style("font-size", "0.7em")
273 .text(function(d) { if (d.type == "domain") {return d.id; } return d.id.substring(0,8); });
274
275 var node = svg.append("g")
276 .attr("class", "nodes")
277 .selectAll("circle")
278 .data(graph.nodes)
279 .enter().append("circle")
280 .attr("r", 10)
281 .attr("fill", function(d) { if (d.root == "true") return color(d.root); return color(d.type); })
282 .call(d3.drag()
283 .on("start", dragstarted)
284 .on("drag", dragged)
285 .on("end", dragended));
286
287 node.on("click",function(d){
288 // console.log("clicked", d.id);
289 // console.log(d);
290 updateInfoBox(d);
291 });
292
293 node.append("title")
294 .text(function(d) { return d.id; });
295
296 simulation
297 .nodes(graph.nodes)
298 .on("tick", ticked);
299
300 simulation.force("link")
301 .links(graph.links);
302
303 function ticked() {
304 link
305 .attr("x1", function(d) { return d.source.x; })
306 .attr("y1", function(d) { return d.source.y; })
307 .attr("x2", function(d) { return d.target.x; })
308 .attr("y2", function(d) { return d.target.y; });
309
310 node
311 .attr("cx", function(d) { return d.x; })
312 .attr("cy", function(d) { return d.y; });
313 text
314 .attr("transform", function(d) { return "translate(" + d.x + "," + d.y + ")"});
315 }
316 createTables();
317 }
318
319
320 function dragstarted(d) {
321 if (!d3.event.active) simulation.alphaTarget(0.3).restart();
322 d.fx = d.x;
323 d.fy = d.y;
324 }
325
326 function dragged(d) {
327 d.fx = d3.event.x;
328 d.fy = d3.event.y;
329 }
330
331 function dragended(d) {
332 if (!d3.event.active) simulation.alphaTarget(0);
333 d.fx = null;
334 d.fy = null;
335 }
336
337 function zoomed() {
338 svg.attr("transform", "translate(" + d3.event.transform.x + "," + d3.event.transform.y + ")" + " scale(" + d3.event.transform.k + ")");
339 }
340
341 d3.select("#generate").on("click", writeDownloadLink);
342 function writeDownloadLink(){
343 try {
344 var isFileSaverSupported = !!new Blob();
345 } catch (e) {
346 alert("blob not supported");
347 }
348
349 var html = d3.select("svg")
350 .attr("title", "graph") //TODO
351 .attr("version", 1.1)
352 .attr("xmlns", "http://www.w3.org/2000/svg")
353 .node().outerHTML;
354
355 var blob = new Blob([html], {type: "image/svg+xml"});
356 saveAs(blob, "certificate_graph.svg"); //TODO root node name
357 };
358
359 function updateInfoBox(d) {
360 if (d) {
361 var el = document.getElementById("node-info");
362 var s = "Type: "+d.type+"</br>";
363 if (d.type == "domain") {
364 s = s + "Domain: "+linkifyDomain(d)+"</br>";
365 s = s + "Status: "+d.status+"</br>";
366 }else if (d.type = "certificate") {
367 s = s + "Hash: "+linkifyCert(d)+"</br>";
368 }
369 el.innerHTML = s;
370 }
371 }
372
373 function createTables() {
374 // TODO: redo this in native d3
375 domainEl = document.getElementById("domain-list");
376 domain_tbody2 = document.createElement('tbody');
377 domain_tbody2.id="domain-list";
378 domainEl.parentNode.replaceChild(domain_tbody2, domainEl);
379
380 certEl = document.getElementById("cert-list");
381 cert_tbody2 = document.createElement('tbody');
382 cert_tbody2.id="cert-list";
383 certEl.parentNode.replaceChild(cert_tbody2, certEl);
384
385 var domainCount = 0;
386 function addTableDomain(d) {
387 //console.log("domain", d);
388 var c = "";
389 if (d.root == "true") {
390 c = "info";
391 }
392 $('#domain-list').append('<tr class="'+c+'"><td>'+ ++domainCount +'</td><td>'+linkifyDomain(d)+'</td><td>'+d.status+'</td><td>'+linkifyAny(d)+'</td></tr>');
393 }
394
395 var certCount = 0;
396 function addTableCert(d) {
397 //console.log("cert", d);
398 $('#cert-list').append('<tr><td>'+ ++certCount + '</td><td>'+linkifyCert(d)+'</td><td>'+linkifyAny(d)+'</td></tr>');
399 }
400
401
402 d3.selectAll('circle').each(function(d){
403 if (d.type == "domain") {
404 addTableDomain(d);
405 }else if (d.type == "certificate") {
406 addTableCert(d);
407 } else {
408 console.log("Unknown Type: ", d.type);
409 }
410 })
411 }
412
413 function linkifyCert(d) {
414 return '<a target="_blank" href="https://crt.sh/?sha256='+d.id+'">'+d.id+'</a>';
415 }
416 function linkifyDomain(d) {
417 return '<a target="_blank" href="https://'+d.id+'">'+d.id+'</a>';
418 }
419 function linkifyAny(d) {
420 return '<a target="_blank" href="https://crt.sh/?q='+d.id+'">&#x1F50E;</a>';
421 }
422
423 function getQueryVariable(variable){
424 var query = window.location.search.substring(1);
425 var vars = query.split("&");
426 for (var i=0;i<vars.length;i++) {
427 var pair = vars[i].split("=");
428 if(pair[0] == variable){return pair[1];}
429 }
430 return "";
431 }
432
433 // new data from url
434 d3.select("#loadURL").on("click", loadURL);
435 function loadURL(){
436 var url = document.getElementById("inputURL").value;
437 history.pushState('', 'CertGraph', "?data="+url);
438 resetGraph();
439 d3.json(url, createGraph);
440 }
441
442 // new data from paste
443 d3.select("#loadPaste").on("click", loadPaste);
444 function loadPaste(){
445 var dataStr = document.getElementById("inputPaste").value;
446 history.pushState('', 'CertGraph', "?");
447 resetGraph();
448 var data = JSON.parse(dataStr);
449 createGraph(null, data);
450 }
451
452 // new data from paste
453 d3.select("#loadFile").on("click", loadFile);
454 function loadFile(){
455 var file = document.getElementById("inputFile").files[0];
456 history.pushState('', 'CertGraph', "?");
457 resetGraph();
458 var reader = new FileReader();
459 reader.onload = function(e) {
460 var dataStr = reader.result;
461 var data = JSON.parse(dataStr);
462 createGraph(null, data);
463 }
464 reader.readAsText(file);
465 }
466
467 var dropbox = document.getElementById('drop-zone');
468 function dragenter(e) {
469 e.stopPropagation();
470 e.preventDefault();
471 dropbox.className = 'upload-drop-zone drop';
472 // console.log("enter");
473 return false;
474 }
475 function dragover(e) {
476 e.stopPropagation();
477 e.preventDefault();
478 // console.log("over");
479 }
480 function dragleave(e) {
481 e.stopPropagation();
482 e.preventDefault();
483 dropbox.className = 'upload-drop-zone';
484 // console.log("leave");
485 return false;
486 }
487 function drop(e) {
488 // console.log("drop");
489 e.stopPropagation();
490 e.preventDefault();
491 dropbox.className = 'upload-drop-zone';
492
493 var dt = e.dataTransfer;
494 var files = dt.files;
495
496 var reader = new FileReader();
497 reader.onload = function(e) {
498 var dataStr = reader.result;
499 var data = JSON.parse(dataStr);
500 resetGraph();
501 createGraph(null, data);
502 }
503 reader.readAsText(files[0]);
504 $('#fileClose').click();
505 return false;
506 }
507 dropbox.addEventListener("dragenter", dragenter, false);
508 dropbox.addEventListener("dragover", dragover, false);
509 dropbox.addEventListener("drop", drop, false);
510 dropbox.addEventListener("dragleave",dragleave, false);
511
512 // load initial graph data
513 var dataURL = getQueryVariable("data");
514 if (dataURL == "") {
515 // default graph
516 dataURL = "https://gist.githubusercontent.com/lanrat/8187d01793bf3e578d76495182654206/raw/c49741b5206d81935febdf563452cc4346381e52/eff.json";
517 }
518 resetGraph();
519 d3.json(dataURL, createGraph);
520 </script>
521 </body>
522 </html>
0 package crtsh
1
2 /*
3 * This file implements an unofficial API client for Comodo's
4 * Certificate Transparency search
5 * https://crt.sh/
6 *
7 * As the API is unofficial and has been reverse engineered it may stop working
8 * at any time and comes with no guarantees.
9 */
10
11 // TODO running in verbose gives error: pq: unnamed prepared statement does not exist
12
13 import (
14 "database/sql"
15 "fmt"
16 "path"
17 "time"
18
19 "github.com/lanrat/certgraph/driver"
20 "github.com/lanrat/certgraph/fingerprint"
21 "github.com/lanrat/certgraph/status"
22 _ "github.com/lib/pq" // portgresql
23 )
24
25 const connStr = "postgresql://[email protected]/certwatch?sslmode=disable"
26 const driverName = "crtsh"
27
28 func init() {
29 driver.AddDriver(driverName)
30 }
31
32 type crtsh struct {
33 db *sql.DB
34 queryLimit int
35 timeout time.Duration
36 save bool
37 savePath string
38 includeSubdomains bool
39 includeExpired bool
40 }
41
42 type crtshCertDriver struct {
43 host string
44 fingerprints driver.FingerprintMap
45 driver *crtsh
46 }
47
48 func (c *crtshCertDriver) GetFingerprints() (driver.FingerprintMap, error) {
49 return c.fingerprints, nil
50 }
51
52 func (c *crtshCertDriver) GetStatus() status.Map {
53 return status.NewMap(c.host, status.New(status.CT))
54 }
55
56 func (c *crtshCertDriver) GetRelated() ([]string, error) {
57 return make([]string, 0), nil
58 }
59
60 func (c *crtshCertDriver) QueryCert(fp fingerprint.Fingerprint) (*driver.CertResult, error) {
61 return c.driver.QueryCert(fp)
62 }
63
64 // Driver creates a new CT driver for crt.sh
65 func Driver(maxQueryResults int, timeout time.Duration, savePath string, includeSubdomains, includeExpired bool) (driver.Driver, error) {
66 d := new(crtsh)
67 d.queryLimit = maxQueryResults
68 d.includeSubdomains = includeSubdomains
69 d.includeExpired = includeExpired
70 var err error
71
72 if len(savePath) > 0 {
73 d.save = true
74 d.savePath = savePath
75 }
76
77 d.db, err = sql.Open("postgres", connStr)
78
79 d.setSQLTimeout(d.timeout.Seconds())
80
81 return d, err
82 }
83
84 func (d *crtsh) GetName() string {
85 return driverName
86 }
87
88 func (d *crtsh) setSQLTimeout(sec float64) error {
89 _, err := d.db.Exec(fmt.Sprintf("SET statement_timeout TO %f;", (1000 * sec)))
90 return err
91 }
92
93 func (d *crtsh) QueryDomain(domain string) (driver.Result, error) {
94 results := &crtshCertDriver{
95 host: domain,
96 fingerprints: make(driver.FingerprintMap),
97 driver: d,
98 }
99
100 queryStr := ""
101
102 if d.includeSubdomains {
103 if d.includeExpired {
104 queryStr = `SELECT digest(certificate.certificate, 'sha256') sha256
105 FROM certificate_identity, certificate
106 WHERE certificate.id = certificate_identity.certificate_id
107 AND (reverse(lower(certificate_identity.name_value)) LIKE reverse(lower('%%.'||$1))
108 OR reverse(lower(certificate_identity.name_value)) LIKE reverse(lower($1)))
109 LIMIT $2`
110 } else {
111 queryStr = `SELECT digest(certificate.certificate, 'sha256') sha256
112 FROM certificate_identity, certificate
113 WHERE certificate.id = certificate_identity.certificate_id
114 AND x509_notAfter(certificate.certificate) > statement_timestamp()
115 AND (reverse(lower(certificate_identity.name_value)) LIKE reverse(lower('%%.'||$1))
116 OR reverse(lower(certificate_identity.name_value)) LIKE reverse(lower($1)))
117 LIMIT $2`
118 }
119 } else {
120 if d.includeExpired {
121 queryStr = `SELECT digest(certificate.certificate, 'sha256') sha256
122 FROM certificate_identity, certificate
123 WHERE certificate.id = certificate_identity.certificate_id
124 AND reverse(lower(certificate_identity.name_value)) LIKE reverse(lower($1))
125 LIMIT $2`
126 } else {
127 queryStr = `SELECT digest(certificate.certificate, 'sha256') sha256
128 FROM certificate_identity, certificate
129 WHERE certificate.id = certificate_identity.certificate_id
130 AND x509_notAfter(certificate.certificate) > statement_timestamp()
131 AND reverse(lower(certificate_identity.name_value)) LIKE reverse(lower($1))
132 LIMIT $2`
133 }
134 }
135
136 if d.includeSubdomains {
137 domain = fmt.Sprintf("%%.%s", domain)
138 }
139
140 try := 0
141 var err error
142 var rows *sql.Rows
143 for try < 5 {
144 // this is a hack while crt.sh gets there stuff togeather
145 try++
146 rows, err = d.db.Query(queryStr, domain, d.queryLimit)
147 if err == nil {
148 break
149 }
150 }
151 /*if try > 1 {
152 fmt.Println("QueryDomain try ", try)
153 }*/
154 if err != nil {
155 return results, err
156 }
157
158 for rows.Next() {
159 var hash []byte
160 err = rows.Scan(&hash)
161 if err != nil {
162 return results, err
163 }
164 results.fingerprints.Add(domain, fingerprint.FromHashBytes(hash))
165 }
166
167 return results, nil
168 }
169
170 func (d *crtsh) QueryCert(fp fingerprint.Fingerprint) (*driver.CertResult, error) {
171 certNode := new(driver.CertResult)
172 certNode.Fingerprint = fp
173 certNode.Domains = make([]string, 0, 5)
174
175 queryStr := `SELECT DISTINCT certificate_identity.name_value
176 FROM certificate, certificate_identity
177 WHERE certificate.id = certificate_identity.certificate_id
178 AND certificate_identity.name_type in ('dNSName', 'commonName')
179 AND digest(certificate.certificate, 'sha256') = $1`
180
181 try := 0
182 var err error
183 var rows *sql.Rows
184 for try < 5 {
185 // this is a hack while crt.sh gets there stuff togeather
186 try++
187 rows, err = d.db.Query(queryStr, fp[:])
188 if err == nil {
189 break
190 }
191 }
192 /*if try > 1 {
193 fmt.Println("QueryCert try ", try)
194 }*/
195 if err != nil {
196 return certNode, err
197 }
198
199 for rows.Next() {
200 var domain string
201 rows.Scan(&domain)
202 certNode.Domains = append(certNode.Domains, domain)
203 }
204
205 if d.save {
206 var rawCert []byte
207 queryStr = `SELECT certificate.certificate
208 FROM certificate
209 WHERE digest(certificate.certificate, 'sha256') = $1`
210 row := d.db.QueryRow(queryStr, fp[:])
211 err = row.Scan(&rawCert)
212 if err != nil {
213 return certNode, err
214 }
215
216 driver.RawCertToPEMFile(rawCert, path.Join(d.savePath, fp.HexString())+".pem")
217 }
218
219 return certNode, nil
220 }
0 package driver
1
2 import (
3 "crypto/x509"
4 "sort"
5 "strings"
6
7 "github.com/lanrat/certgraph/fingerprint"
8 "github.com/lanrat/certgraph/status"
9 )
10
11 // Drivers contains all the drivers that have been registered
12 var Drivers []string
13
14 // AddDriver should be called in the init() function of every driver to register them here
15 func AddDriver(name string) {
16 Drivers = append(Drivers, name)
17 }
18
19 // Driver is a universal unifying interface to support CT, http and much more!
20 type Driver interface {
21 // QueryDomain is the main entrypoint for Driver Searching
22 // The domain provided will return a CertDriver instance which can be used to query the
23 // certificates for the provided domain using the driver
24 QueryDomain(domain string) (Result, error)
25
26 // GetName returns the name of the driver
27 GetName() string
28 }
29
30 // Result is a sub-driver that allows querying certificate details from a previously queried domain
31 type Result interface {
32 // GetStatus returns the status of the initial domain queried with the Driver.QueryDomain call
33 GetStatus() status.Map
34
35 // returns a list of additional related domains discovered while looking up the provided domain
36 GetRelated() ([]string, error)
37
38 // GetFingerprints returns an array of the certificate fingerprints associated with the Domain
39 // pass return fingerprints to QueryCert to get certificate details
40 GetFingerprints() (FingerprintMap, error)
41
42 // QueryCert returns the details of the provided certificate or an error if not found
43 QueryCert(fp fingerprint.Fingerprint) (*CertResult, error)
44 }
45
46 // FingerprintMap stores a mapping of domains to Fingerprints returned from the driver
47 // in the case where multiple domains where queries (redirects, related, etc..) the
48 // matching certificates will be in this map
49 // the fingerprints returned are guaranteed to be a complete result for the domain's certs, but related domains may or may not be complete
50 type FingerprintMap map[string][]fingerprint.Fingerprint
51
52 // Add adds a domain and fingerprint to the map
53 func (f FingerprintMap) Add(domain string, fp fingerprint.Fingerprint) {
54 f[domain] = append(f[domain], fp)
55 }
56
57 // CertResult is an object to hold the fingerprint and Domains for a returned certificate
58 type CertResult struct {
59 Fingerprint fingerprint.Fingerprint
60 Domains []string
61 }
62
63 // NewCertResult creates a new CertResult struct from an x509 cert
64 func NewCertResult(cert *x509.Certificate) *CertResult {
65 certResult := new(CertResult)
66
67 // generate Fingerprint
68 certResult.Fingerprint = fingerprint.FromBytes(cert.Raw)
69
70 // domains
71 // used to ensure uniq entries in domains array
72 domainMap := make(map[string]bool)
73 // add the CommonName just to be safe
74 cn := strings.ToLower(cert.Subject.CommonName)
75 if len(cn) > 0 {
76 domainMap[cn] = true
77 }
78 for _, domain := range cert.DNSNames {
79 if len(domain) > 0 {
80 domain = strings.ToLower(domain)
81 domainMap[domain] = true
82 }
83 }
84 for domain := range domainMap {
85 certResult.Domains = append(certResult.Domains, domain)
86 }
87 sort.Strings(certResult.Domains)
88
89 return certResult
90 }
0 package driver
1
2 import "fmt"
3
4 // Example provides a simple entrypoint to test a driver on an individual domain
5 func Example(domain string, driver Driver) error {
6 certDriver, err := driver.QueryDomain(domain)
7 if err != nil {
8 return err
9 }
10
11 relatedDomains, err := certDriver.GetRelated()
12 if err != nil {
13 return err
14 }
15 if len(relatedDomains) > 0 {
16 fmt.Printf("Related:\n")
17 }
18 for _, relatedDomain := range relatedDomains {
19 fmt.Printf("\t%s\n", relatedDomain)
20 }
21
22 fingerprintMap, err := certDriver.GetFingerprints()
23 if err != nil {
24 return err
25 }
26 for domain, fingerprints := range fingerprintMap {
27 for i := range fingerprints {
28 fmt.Printf("%s: %s\n", domain, fingerprints[i].HexString())
29 cert, err := certDriver.QueryCert(fingerprints[i])
30 if err != nil {
31 return err
32 }
33 for j := range cert.Domains {
34 fmt.Println("\t", cert.Domains[j])
35 }
36 }
37 }
38
39 return nil
40 }
0 package google
1
2 /*
3 * This file implements an unofficial API client for Google's
4 * Certificate Transparency search
5 * https://transparencyreport.google.com/https/certificates
6 *
7 * As the API is unofficial and has been reverse engineered it may stop working
8 * at any time and comes with no guarantees.
9 */
10
11 import (
12 "encoding/json"
13 "errors"
14 "io/ioutil"
15 "net/http"
16 "net/url"
17 "strconv"
18 "time"
19
20 "github.com/lanrat/certgraph/driver"
21 "github.com/lanrat/certgraph/fingerprint"
22 "github.com/lanrat/certgraph/status"
23 )
24
25 const driverName = "google"
26
27 func init() {
28 driver.AddDriver(driverName)
29 }
30
31 // Base URLs for Google's CT API
32 const searchURL1 = "https://transparencyreport.google.com/transparencyreport/api/v3/httpsreport/ct/certsearch?include_expired=false&include_subdomains=false&domain=example.com"
33 const searchURL2 = "https://transparencyreport.google.com/transparencyreport/api/v3/httpsreport/ct/certsearch/page?p=DEADBEEF"
34 const certURL = "https://transparencyreport.google.com/transparencyreport/api/v3/httpsreport/ct/certbyhash?hash=DEADBEEF"
35
36 type googleCT struct {
37 maxPages float64 // this is a float because that is the type automatically decoded from the JSON response
38 jsonClient *http.Client
39 includeExpired bool
40 includeSubdomains bool
41 }
42
43 type googleCertDriver struct {
44 host string
45 fingerprints driver.FingerprintMap
46 driver *googleCT
47 }
48
49 func (c *googleCertDriver) GetFingerprints() (driver.FingerprintMap, error) {
50 return c.fingerprints, nil
51 }
52
53 func (c *googleCertDriver) GetStatus() status.Map {
54 return status.NewMap(c.host, status.New(status.CT))
55 }
56
57 func (c *googleCertDriver) GetRelated() ([]string, error) {
58 return make([]string, 0), nil
59 }
60
61 func (c *googleCertDriver) QueryCert(fp fingerprint.Fingerprint) (*driver.CertResult, error) {
62 return c.driver.QueryCert(fp)
63 }
64
65 // Driver creates a new CT driver for google
66 func Driver(maxQueryPages int, savePath string, includeSubdomains, includeExpired bool) (driver.Driver, error) {
67 d := new(googleCT)
68 d.maxPages = float64(maxQueryPages)
69 d.jsonClient = &http.Client{Timeout: 10 * time.Second}
70 d.includeExpired = includeExpired
71 d.includeSubdomains = includeSubdomains
72
73 if len(savePath) > 0 {
74 return d, errors.New("google driver does not support saving")
75
76 }
77
78 return d, nil
79 }
80
81 func (d *googleCT) GetName() string {
82 return driverName
83 }
84
85 // getJsonP gets JSON from url and parses it into target object
86 func (d *googleCT) getJSONP(url string, target interface{}) error {
87 r, err := d.jsonClient.Get(url)
88 if err != nil {
89 return err
90 }
91 defer r.Body.Close()
92 if r.StatusCode != http.StatusOK {
93 return errors.New("Got non OK HTTP status: '" + r.Status + "' on URL: " + url)
94 }
95
96 respData, err := ioutil.ReadAll(r.Body)
97 if err != nil {
98 return err
99 }
100
101 respData = respData[5:] // this removes the leading ")]}'" from the response
102
103 return json.Unmarshal(respData, target)
104 }
105
106 func (d *googleCT) QueryDomain(domain string) (driver.Result, error) {
107 results := &googleCertDriver{
108 fingerprints: make(driver.FingerprintMap),
109 driver: d,
110 host: domain,
111 }
112
113 u, err := url.Parse(searchURL1)
114 if err != nil {
115 return results, err
116 }
117
118 // get page 1
119 q := u.Query()
120 q.Set("include_expired", strconv.FormatBool(d.includeExpired))
121 q.Set("include_subdomains", strconv.FormatBool(d.includeSubdomains))
122 q.Set("domain", domain)
123 u.RawQuery = q.Encode()
124
125 var raw [][]interface{}
126 nextURL := u.String()
127 currentPage := float64(1)
128
129 // TODO allow for selective pagination
130
131 // iterate over results
132 for len(nextURL) > 1 && currentPage <= d.maxPages {
133 err = d.getJSONP(nextURL, &raw)
134 if err != nil {
135 return results, err
136 }
137
138 // simple corectness checks
139 if raw[0][0] != "https.ct.cdsr" {
140 return results, errors.New("Got Unexpected Query output: " + raw[0][0].(string))
141 }
142 if len(raw[0]) != 4 {
143 // result not correct length, likely no results
144 //fmt.Println(raw[0])
145 break
146 }
147 if len(raw[0][3].([]interface{})) != 5 {
148 // pageinfo result not correct length, likely no results
149 //fmt.Println(raw[0])
150 break
151 }
152
153 // pageInfo: [prevToken, nextToken, ? currentPage, totalPages]
154 pageInfo := raw[0][3].([]interface{})
155 currentPage = pageInfo[3].(float64)
156
157 foundCerts := raw[0][1].([]interface{})
158 for _, foundCert := range foundCerts {
159 certHash := foundCert.([]interface{})[5].(string)
160 certFP := fingerprint.FromB64(certHash)
161 results.fingerprints.Add(domain, certFP)
162 }
163 //fmt.Println("Page:", pageInfo[3])
164
165 // create url or next page
166 nextURL = ""
167 if pageInfo[1] != nil {
168 u, err := url.Parse(searchURL2)
169 if err != nil {
170 return results, err
171 }
172
173 // get page n
174 q := u.Query()
175 q.Set("p", pageInfo[1].(string))
176 u.RawQuery = q.Encode()
177 nextURL = u.String()
178 }
179 }
180
181 return results, nil
182 }
183
184 func (d *googleCT) QueryCert(fp fingerprint.Fingerprint) (*driver.CertResult, error) {
185 certNode := new(driver.CertResult)
186 certNode.Fingerprint = fp
187 certNode.Domains = make([]string, 0, 5)
188
189 u, err := url.Parse(certURL)
190 if err != nil {
191 return certNode, err
192 }
193
194 q := u.Query()
195 q.Set("hash", fp.B64Encode())
196 u.RawQuery = q.Encode()
197
198 var raw [][]interface{}
199
200 err = d.getJSONP(u.String(), &raw)
201 if err != nil {
202 return certNode, err
203 }
204
205 // simple corectness checks
206 if raw[0][0] != "https.ct.chr" {
207 return certNode, errors.New("Got Unexpected Cert output: " + raw[0][0].(string))
208 }
209 if len(raw[0]) != 3 {
210 // result not correct length, likely no results
211 //fmt.Println(raw[0])
212 return certNode, errors.New("Cert Does not exist! output: " + raw[0][0].(string))
213 }
214
215 certInfo := raw[0][1].([]interface{})
216 domains := certInfo[7].([]interface{})
217
218 for _, domain := range domains {
219 certNode.Domains = append(certNode.Domains, domain.(string))
220 }
221
222 return certNode, nil
223 }
0 package http
1
2 import (
3 "crypto/tls"
4 "fmt"
5 "net"
6 "net/http"
7 "path"
8 "time"
9
10 "github.com/lanrat/certgraph/driver"
11 "github.com/lanrat/certgraph/fingerprint"
12 "github.com/lanrat/certgraph/status"
13 )
14
15 const driverName = "http"
16
17 func init() {
18 driver.AddDriver(driverName)
19 }
20
21 type httpDriver struct {
22 port string
23 save bool
24 savePath string
25 tlsConfig *tls.Config
26 timeout time.Duration
27 }
28
29 type httpCertDriver struct {
30 parent *httpDriver
31 client *http.Client
32 fingerprints driver.FingerprintMap
33 status status.Map
34 related []string
35 certs map[fingerprint.Fingerprint]*driver.CertResult
36 }
37
38 func (c *httpCertDriver) GetFingerprints() (driver.FingerprintMap, error) {
39 return c.fingerprints, nil
40 }
41
42 func (c *httpCertDriver) GetStatus() status.Map {
43 return c.status
44 }
45
46 func (c *httpCertDriver) GetRelated() ([]string, error) {
47 return c.related, nil
48 }
49
50 func (c *httpCertDriver) QueryCert(fp fingerprint.Fingerprint) (*driver.CertResult, error) {
51 cert, found := c.certs[fp]
52 if found {
53 return cert, nil
54 }
55 return nil, fmt.Errorf("Certificate with Fingerprint %s not found", fp.HexString())
56 }
57
58 // Driver creates a new SSL driver for HTTP Connections
59 func Driver(timeout time.Duration, savePath string) (driver.Driver, error) {
60 d := new(httpDriver)
61 d.port = "443"
62 if len(savePath) > 0 {
63 d.save = true
64 d.savePath = savePath
65 }
66 d.timeout = timeout
67 d.tlsConfig = &tls.Config{
68 InsecureSkipVerify: true,
69 }
70
71 return d, nil
72 }
73
74 func (d *httpDriver) GetName() string {
75 return driverName
76 }
77
78 func (d *httpDriver) newHTTPCertDriver() *httpCertDriver {
79 result := &httpCertDriver{
80 parent: d,
81 status: make(status.Map),
82 fingerprints: make(driver.FingerprintMap),
83 certs: make(map[fingerprint.Fingerprint]*driver.CertResult),
84 }
85 // set client & client.Transport separately so that dialTLS checkRedirect can be referenced
86 result.client = &http.Client{
87 Timeout: d.timeout,
88 CheckRedirect: result.checkRedirect,
89 }
90 result.client.Transport = &http.Transport{
91 TLSClientConfig: d.tlsConfig,
92 TLSHandshakeTimeout: d.timeout,
93 ResponseHeaderTimeout: d.timeout,
94 ExpectContinueTimeout: d.timeout,
95 DialTLS: result.dialTLS,
96 }
97 return result
98 }
99
100 // GetCert gets the certificates found for a given domain
101 func (d *httpDriver) QueryDomain(host string) (driver.Result, error) {
102 results := d.newHTTPCertDriver()
103
104 resp, err := results.client.Get(fmt.Sprintf("https://%s", host))
105 fullStatus := status.CheckNetErr(err)
106 if fullStatus != status.GOOD {
107 return results, err // in some rare cases this error can be ignored
108 }
109 defer resp.Body.Close()
110
111 // set final domain status
112 results.status.Set(resp.Request.URL.Hostname(), status.New(status.GOOD))
113 // no need to add certificate to c.certs and c.fingerprints here, handled in dialTLS method
114 return results, nil
115 }
116
117 // only called after a redirect is detected
118 // req has the next request to send, via has the last requests
119 // not called for the first HTTP request that replied with the initial redirect
120 func (c *httpCertDriver) checkRedirect(req *http.Request, via []*http.Request) error {
121 //fmt.Printf("Redirect %s -> %s\n", via[0].URL, req.URL)
122 // set both domain's status's
123 c.status.Set(via[0].URL.Hostname(), status.NewMeta(status.REDIRECT, req.URL.Hostname()))
124 c.status.Set(req.URL.Hostname(), status.New(status.UNKNOWN))
125 c.related = append(c.related, req.URL.Hostname())
126 if len(via) >= 10 { // stop after 10 redirects
127 // this stops the redirect
128 return http.ErrUseLastResponse
129 }
130 return nil
131 }
132
133 func (c *httpCertDriver) dialTLS(network, addr string) (net.Conn, error) {
134 dialer := &net.Dialer{Timeout: c.client.Timeout}
135 conn, err := tls.DialWithDialer(dialer, network, addr, c.parent.tlsConfig)
136 if conn == nil {
137 return conn, err
138 }
139 // get certs passing by
140 connState := conn.ConnectionState()
141
142 // only look at leaf certificate which is valid for domain, rest of cert chain is ignored
143 certResult := driver.NewCertResult(connState.PeerCertificates[0])
144 c.certs[certResult.Fingerprint] = certResult
145 host, _, err := net.SplitHostPort(addr)
146 if err != nil {
147 return conn, err
148 }
149 c.fingerprints.Add(host, certResult.Fingerprint)
150
151 // save
152 if c.parent.save && len(connState.PeerCertificates) > 0 {
153 driver.CertsToPEMFile(connState.PeerCertificates, path.Join(c.parent.savePath, certResult.Fingerprint.HexString())+".pem")
154 }
155
156 return conn, err
157 }
0 package driver
1
2 import (
3 "crypto/x509"
4 "encoding/pem"
5 "os"
6 )
7
8 // CertsToPEMFile saves certificates to local pem file
9 func CertsToPEMFile(certs []*x509.Certificate, file string) error {
10 if fileExists(file) {
11 return nil
12 }
13 f, err := os.Create(file)
14 if err != nil {
15 return err
16 }
17 defer f.Close()
18 for _, cert := range certs {
19 pem.Encode(f, &pem.Block{Type: "CERTIFICATE", Bytes: cert.Raw})
20 }
21 return nil
22 }
23
24 // RawCertToPEMFile saves raw certificate to local pem file
25 func RawCertToPEMFile(cert []byte, file string) error {
26 if fileExists(file) {
27 return nil
28 }
29 f, err := os.Create(file)
30 if err != nil {
31 return err
32 }
33 defer f.Close()
34 pem.Encode(f, &pem.Block{Type: "CERTIFICATE", Bytes: cert})
35 return nil
36 }
37
38 func fileExists(f string) bool {
39 _, err := os.Stat(f)
40 if os.IsNotExist(err) {
41 return false
42 }
43 return err == nil
44 }
0 package smtp
1
2 import (
3 "context"
4 "crypto/tls"
5 "crypto/x509"
6 "fmt"
7 "net"
8 "net/smtp"
9 "path"
10 "strings"
11 "time"
12
13 "github.com/lanrat/certgraph/driver"
14 "github.com/lanrat/certgraph/fingerprint"
15 "github.com/lanrat/certgraph/status"
16 )
17
18 const driverName = "smtp"
19
20 func init() {
21 driver.AddDriver(driverName)
22 }
23
24 type smtpDriver struct {
25 port string
26 save bool
27 savePath string
28 tlsConfig *tls.Config
29 timeout time.Duration
30 }
31
32 type smtpCertDriver struct {
33 host string
34 fingerprints driver.FingerprintMap
35 status status.Map
36 mx []string
37 certs map[fingerprint.Fingerprint]*driver.CertResult
38 }
39
40 func (c *smtpCertDriver) GetFingerprints() (driver.FingerprintMap, error) {
41 return c.fingerprints, nil
42 }
43
44 func (c *smtpCertDriver) GetStatus() status.Map {
45 return c.status
46 }
47
48 func (c *smtpCertDriver) GetRelated() ([]string, error) {
49 return c.mx, nil
50 }
51
52 func (c *smtpCertDriver) QueryCert(fp fingerprint.Fingerprint) (*driver.CertResult, error) {
53 cert, found := c.certs[fp]
54 if found {
55 return cert, nil
56 }
57 return nil, fmt.Errorf("Certificate with Fingerprint %s not found", fp.HexString())
58 }
59
60 // Driver creates a new SSL driver for SMTP Connections
61 func Driver(timeout time.Duration, savePath string) (driver.Driver, error) {
62 d := new(smtpDriver)
63 d.port = "25"
64 if len(savePath) > 0 {
65 d.save = true
66 d.savePath = savePath
67 }
68 d.tlsConfig = &tls.Config{
69 InsecureSkipVerify: true,
70 }
71 d.timeout = timeout
72
73 return d, nil
74 }
75
76 func (d *smtpDriver) GetName() string {
77 return driverName
78 }
79
80 func (d *smtpDriver) smtpGetCerts(host string) ([]*x509.Certificate, error) {
81 var certs []*x509.Certificate
82 addr := net.JoinHostPort(host, d.port)
83 dialer := &net.Dialer{Timeout: d.timeout}
84
85 conn, err := dialer.Dial("tcp", addr)
86 if err != nil {
87 return certs, err
88 }
89 defer conn.Close()
90 smtp, err := smtp.NewClient(conn, host)
91 if err != nil {
92 return certs, err
93 }
94 err = smtp.StartTLS(d.tlsConfig)
95 if err != nil {
96 return certs, err
97 }
98 connState, ok := smtp.TLSConnectionState()
99 if !ok {
100 return certs, err
101 }
102 return connState.PeerCertificates, err
103 }
104
105 // QueryDomain gets the certificates found for a given domain
106 func (d *smtpDriver) QueryDomain(host string) (driver.Result, error) {
107 results := &smtpCertDriver{
108 host: host,
109 status: make(status.Map),
110 fingerprints: make(driver.FingerprintMap),
111 certs: make(map[fingerprint.Fingerprint]*driver.CertResult),
112 }
113
114 // get related in different query
115 results.mx, _ = d.getMX(host)
116
117 certs, err := d.smtpGetCerts(host)
118 smtpStatus := status.CheckNetErr(err)
119 metaStatus := ""
120 if len(results.mx) > 0 {
121 metaStatus = fmt.Sprintf("MX(%s)", strings.Join(results.mx, " "))
122 }
123 results.status.Set(host, status.NewMeta(smtpStatus, metaStatus))
124
125 if smtpStatus != status.GOOD {
126 return results, nil
127 }
128
129 // only look at leaf certificate which is valid for domain, rest of cert chain is ignored
130 certResult := driver.NewCertResult(certs[0])
131 results.certs[certResult.Fingerprint] = certResult
132 results.fingerprints.Add(host, certResult.Fingerprint)
133
134 // save
135 if d.save && len(certs) > 0 {
136 driver.CertsToPEMFile(certs, path.Join(d.savePath, certResult.Fingerprint.HexString())+".pem")
137 }
138
139 return results, nil
140 }
141
142 // getMX returns the MX records for the provided domain
143 func (d *smtpDriver) getMX(domain string) ([]string, error) {
144 domains := make([]string, 0, 5)
145 ctx, cancel := context.WithTimeout(context.Background(), d.timeout)
146 defer cancel()
147 mx, err := net.DefaultResolver.LookupMX(ctx, domain)
148 if err != nil {
149 return domains, err
150 }
151 for _, v := range mx {
152 domains = append(domains, strings.TrimSuffix(v.Host, "."))
153 }
154 return domains, nil
155 }
0 package fingerprint
1
2 import (
3 "crypto/sha256"
4 "encoding/base64"
5 "fmt"
6 )
7
8 // Fingerprint sha256 of certificate bytes
9 type Fingerprint [sha256.Size]byte
10
11 // HexString print Fingerprint as hex
12 func (fp *Fingerprint) HexString() string {
13 return fmt.Sprintf("%X", *fp)
14 }
15
16 // FromHashBytes returns a Fingerprint generated by the first len(Fingerprint) bytes
17 func FromHashBytes(data []byte) Fingerprint {
18 var fp Fingerprint
19 /*if len(data) != sha256.Size {
20 v("Data is not correct SHA256 size", data)
21 }*/
22 for i := 0; i < len(data) && i < len(fp); i++ {
23 fp[i] = data[i]
24 }
25 return fp
26 }
27
28 // FromBytes returns a Fingerprint generated by the provided bytes
29 func FromBytes(data []byte) Fingerprint {
30 var fp Fingerprint
31 fp = sha256.Sum256(data)
32 return fp
33 }
34
35 // FromB64 returns a Fingerprint from a base64 encoded hash string
36 func FromB64(hash string) Fingerprint {
37 data, _ := base64.StdEncoding.DecodeString(hash)
38 /*if err != nil {
39 fmt.Println(err)
40 }*/
41 return FromHashBytes(data)
42 }
43
44 // B64Encode returns the b64 string of a Fingerprint
45 func (fp *Fingerprint) B64Encode() string {
46 return base64.StdEncoding.EncodeToString(fp[:])
47 }
0 module github.com/lanrat/certgraph
1
2 require (
3 github.com/lib/pq v1.0.0
4 github.com/weppos/publicsuffix-go v0.4.0
5 golang.org/x/net v0.0.0-20180906233101-161cd47e91fd // indirect
6 golang.org/x/text v0.3.0 // indirect
7 )
0 package graph
1
2 import (
3 "fmt"
4 "strings"
5
6 "github.com/lanrat/certgraph/dns"
7 "github.com/lanrat/certgraph/fingerprint"
8 )
9
10 // CertNode graph node to store certificate information
11 type CertNode struct {
12 Fingerprint fingerprint.Fingerprint
13 Domains []string
14 foundMap map[string]bool
15 }
16
17 func (c *CertNode) String() string {
18 return fmt.Sprintf("%s\t%s\t%v", c.Fingerprint.HexString(), c.Found(), c.Domains)
19 }
20
21 // Found returns a list of drivers that found this cert
22 func (c *CertNode) Found() []string {
23 found := make([]string, 0, len(c.foundMap))
24 for i := range c.foundMap {
25 found = append(found, i)
26 }
27 return found
28 }
29
30 // AddFound adds a driver name to the source of the certificate
31 func (c *CertNode) AddFound(driver string) {
32 if c.foundMap == nil {
33 c.foundMap = make(map[string]bool)
34 }
35 c.foundMap[driver] = true
36 }
37
38 // CDNCert returns true if we think the certificate belongs to a CDN
39 // very weak detection, only supports fastly & cloudflare
40 func (c *CertNode) CDNCert() bool {
41 for _, domain := range c.Domains {
42 // cloudflare
43 if strings.HasSuffix(domain, ".cloudflaressl.com") {
44 return true
45 }
46 // fastly
47 if strings.HasSuffix(domain, "fastly.net") {
48 return true
49 }
50 // akamai
51 if strings.HasSuffix(domain, ".akamai.net") {
52 return true
53 }
54
55 }
56 return false
57 }
58
59 // TLDPlus1Count the number of tld+1 domains in the certificate
60 func (c *CertNode) TLDPlus1Count() int {
61 tldPlus1Domains := make(map[string]bool)
62 for _, domain := range c.Domains {
63 tldPlus1, err := dns.TLDPlus1(domain)
64 if err != nil {
65 continue
66 }
67 tldPlus1Domains[tldPlus1] = true
68 }
69 return len(tldPlus1Domains)
70 }
71
72 // ToMap returns a map of the CertNode's fields (weak serialization)
73 func (c *CertNode) ToMap() map[string]string {
74 m := make(map[string]string)
75 m["type"] = "certificate"
76 m["id"] = c.Fingerprint.HexString()
77 m["found"] = strings.Join(c.Found(), " ")
78 return m
79 }
0 package graph
1
2 import (
3 "fmt"
4 "strconv"
5 "strings"
6 "time"
7
8 "github.com/lanrat/certgraph/dns"
9 "github.com/lanrat/certgraph/fingerprint"
10 "github.com/lanrat/certgraph/status"
11 )
12
13 // DomainNode structure to store a domain and its edges
14 type DomainNode struct {
15 Domain string
16 Depth uint
17 Certs map[fingerprint.Fingerprint][]string
18 RelatedDomains status.Map
19 Status status.Status
20 Root bool
21 HasDNS bool
22 }
23
24 // NewDomainNode constructor for DomainNode, converts domain to nonWildcard
25 func NewDomainNode(domain string, depth uint) *DomainNode {
26 domainNode := new(DomainNode)
27 domainNode.Domain = nonWildcard(domain)
28 domainNode.Depth = depth
29 domainNode.Certs = make(map[fingerprint.Fingerprint][]string)
30 domainNode.RelatedDomains = make(status.Map)
31 return domainNode
32 }
33
34 // AddRelatedDomains adds the domains in the provided array to the domainNode's
35 // related domains status map with an unknown status if they are not already
36 // in the map
37 func (d *DomainNode) AddRelatedDomains(domains []string) {
38 for _, domain := range domains {
39 if _, ok := d.RelatedDomains[domain]; ok {
40 continue
41 }
42 d.RelatedDomains[domain] = status.New(status.UNKNOWN)
43 }
44 }
45
46 // CheckForDNS checks for the existence of DNS records for the domain's tld+1
47 // sets the value to the node and returns the result as well
48 func (d *DomainNode) CheckForDNS(timeout time.Duration) (bool, error) {
49 hasDNS, err := dns.HasRecordsCache(d.Domain, timeout)
50
51 d.HasDNS = hasDNS
52 return hasDNS, err
53 }
54
55 // AddStatusMap adds the status' in the map to the DomainNode
56 // also sets the Node's own status if it is in the Map
57 // side effect: will delete its own status from the provided map
58 func (d *DomainNode) AddStatusMap(m status.Map) {
59 if status, ok := m[d.Domain]; ok {
60 d.Status = status
61 delete(m, d.Domain)
62 }
63 for domain, status := range m {
64 d.RelatedDomains[domain] = status
65 }
66 }
67
68 // GetCertificates returns a list of known certificate fingerprints for the domain
69 func (d *DomainNode) GetCertificates() []fingerprint.Fingerprint {
70 fingerprints := make([]fingerprint.Fingerprint, 0, len(d.Certs))
71 for fingerprint := range d.Certs {
72 fingerprints = append(fingerprints, fingerprint)
73 }
74 return fingerprints
75 }
76
77 // get the string representation of a node
78 func (d *DomainNode) String() string {
79 certString := ""
80 // Certs
81 if len(d.Certs) > 0 {
82 for fingerprint := range d.Certs {
83 certString = fmt.Sprintf("%s %s", certString, fingerprint.HexString())
84 }
85 }
86 return fmt.Sprintf("%s\t%d\t%s\t%s", d.Domain, d.Depth, d.Status.String(), certString)
87 }
88
89 // AddCertFingerprint appends a Fingerprint to the DomainNode's list of certificates
90 func (d *DomainNode) AddCertFingerprint(fp fingerprint.Fingerprint, certSource string) {
91 d.Certs[fp] = append(d.Certs[fp], certSource)
92 }
93
94 // ToMap returns a map of the DomainNode's fields (weak serialization)
95 func (d *DomainNode) ToMap() map[string]string {
96 related := make([]string, 0, len(d.RelatedDomains))
97 for domain := range d.RelatedDomains {
98 related = append(related, domain)
99 }
100 relatedString := strings.Join(related, " ")
101 m := make(map[string]string)
102 m["type"] = "domain"
103 m["id"] = d.Domain
104 m["status"] = d.Status.String()
105 m["root"] = strconv.FormatBool(d.Root)
106 m["depth"] = strconv.FormatUint(uint64(d.Depth), 10)
107 m["related"] = relatedString
108 m["hasDNS"] = strconv.FormatBool(d.HasDNS)
109 return m
110 }
0 package graph
1
2 import (
3 "strings"
4 "sync"
5
6 "github.com/lanrat/certgraph/fingerprint"
7 )
8
9 // CertGraph main graph storage engine
10 type CertGraph struct {
11 domains sync.Map
12 certs sync.Map
13 numDomains int
14 depth uint
15 }
16
17 // NewCertGraph instantiates a new empty CertGraph
18 func NewCertGraph() *CertGraph {
19 graph := new(CertGraph)
20 return graph
21 }
22
23 // LoadOrStoreCert will return the CertNode in the graph with the provided node's fingerprint, or store the node if it did not already exist
24 // returned bool is true if the CertNode was found, false if stored
25 func (graph *CertGraph) LoadOrStoreCert(certNode *CertNode) (*CertNode, bool) {
26 foundCertNode, ok := graph.certs.LoadOrStore(certNode.Fingerprint, certNode)
27 return foundCertNode.(*CertNode), ok
28 }
29
30 // AddCert add a CertNode to the graph
31 func (graph *CertGraph) AddCert(certNode *CertNode) {
32 // save the cert to the graph
33 // if it already exists we overwrite, it is simpler than checking first.
34 graph.certs.Store(certNode.Fingerprint, certNode)
35 }
36
37 // AddDomain add a DomainNode to the graph
38 func (graph *CertGraph) AddDomain(domainNode *DomainNode) {
39 graph.numDomains++
40 // save the new maximum depth if greather then current
41 if domainNode.Depth > graph.depth {
42 graph.depth = domainNode.Depth
43 }
44 // save the domain to the graph
45 // if it already exists we overwrite, it is simpler than checking first.
46 // graph.numDomains should still be accurate because we only call this after checking that we have not visited the node before.
47 graph.domains.Store(domainNode.Domain, domainNode)
48 }
49
50 //NumDomains returns the number of domains in the graph
51 func (graph *CertGraph) NumDomains() int {
52 return graph.numDomains
53 }
54
55 //DomainDepth returns the maximum depth of the graph from the initial root domains
56 func (graph *CertGraph) DomainDepth() uint {
57 return graph.depth
58 }
59
60 // GetCert returns (CertNode, found) for the certificate with the provided Fingerprint in the graph if found
61 func (graph *CertGraph) GetCert(fp fingerprint.Fingerprint) (*CertNode, bool) {
62 node, ok := graph.certs.Load(fp)
63 if ok {
64 return node.(*CertNode), true
65 }
66 return nil, false
67 }
68
69 // GetDomain returns (DomainNode, found) for the domain in the graph if found
70 func (graph *CertGraph) GetDomain(domain string) (*DomainNode, bool) {
71 node, ok := graph.domains.Load(domain)
72 if ok {
73 return node.(*DomainNode), true
74 }
75 return nil, false
76 }
77
78 // GetDomainNeighbors given a domain, return the list of all other domains that share a certificate with the provided domain that are in the graph
79 // cdn will include CDN certs as well
80 func (graph *CertGraph) GetDomainNeighbors(domain string, cdn bool, maxSANsSize int) []string {
81 neighbors := make(map[string]bool)
82
83 domain = nonWildcard(domain)
84 node, ok := graph.domains.Load(domain)
85 if ok {
86 domainNode := node.(*DomainNode)
87 // related cert neighbors
88 for relatedDomain := range domainNode.RelatedDomains {
89 neighbors[relatedDomain] = true
90 }
91
92 // Cert neighbors
93 for _, fp := range domainNode.GetCertificates() {
94 node, ok := graph.certs.Load(fp)
95 if ok {
96 certNode := node.(*CertNode)
97 if !cdn && certNode.CDNCert() {
98 //v(domain, "-> CDN CERT")
99 } else if maxSANsSize > 0 && certNode.TLDPlus1Count() > maxSANsSize {
100 //v(domain, "-> Large CERT")
101 } else {
102 for _, neighbor := range certNode.Domains {
103 neighbors[neighbor] = true
104 //v(domain, "-- CT -->", neighbor)
105 }
106 }
107 }
108 }
109 }
110
111 //exclude domain from own neighbors list
112 neighbors[domain] = false
113
114 // convert map to array
115 neighborList := make([]string, 0, len(neighbors))
116 for key := range neighbors {
117 if neighbors[key] {
118 neighborList = append(neighborList, key)
119 }
120 }
121 return neighborList
122 }
123
124 // GenerateMap returns a map representation of the certificate graph
125 // used for JSON serialization
126 func (graph *CertGraph) GenerateMap() map[string]interface{} {
127 m := make(map[string]interface{})
128 nodes := make([]map[string]string, 0, 2*graph.numDomains)
129 links := make([]map[string]string, 0, 2*graph.numDomains)
130
131 // add all domain nodes
132 graph.domains.Range(func(key, value interface{}) bool {
133 domainNode := value.(*DomainNode)
134 nodes = append(nodes, domainNode.ToMap())
135 for fingerprint, found := range domainNode.Certs {
136 links = append(links, map[string]string{"source": domainNode.Domain, "target": fingerprint.HexString(), "type": strings.Join(found, " ")})
137 }
138 return true
139 })
140
141 // add all cert nodes
142 graph.certs.Range(func(key, value interface{}) bool {
143 certNode := value.(*CertNode)
144 nodes = append(nodes, certNode.ToMap())
145 for _, domain := range certNode.Domains {
146 domain = nonWildcard(domain)
147 _, ok := graph.GetDomain(domain)
148 if ok {
149 links = append(links, map[string]string{"source": certNode.Fingerprint.HexString(), "target": domain, "type": "sans"})
150 }
151 }
152 return true
153 })
154
155 m["nodes"] = nodes
156 m["links"] = links
157 m["depth"] = graph.depth
158 m["numDomains"] = graph.numDomains
159 return m
160 }
0 package graph
1
2 import (
3 "strings"
4 )
5
6 // given a domain returns the non-wildcard version of that domain
7 func nonWildcard(domain string) string {
8 return strings.TrimPrefix(domain, "*.")
9 }
0 package status
1
2 import (
3 "fmt"
4 "net"
5 "syscall"
6 )
7
8 // DomainStatus domain node connection status
9 type DomainStatus int
10
11 // Status holds the domain status and optionally more information
12 // ex: redirects will have the redirected domain in Meta
13 type Status struct {
14 Status DomainStatus
15 Meta string
16 }
17
18 // New returns a new Status object with the provided DomainStatus
19 func New(domainStatus DomainStatus) Status {
20 return Status{
21 Status: domainStatus,
22 }
23 }
24
25 // NewMeta returns a new Status with the provied meta
26 func NewMeta(domainStatus DomainStatus, meta string) Status {
27 s := New(domainStatus)
28 s.Meta = meta
29 return s
30 }
31
32 func (s *Status) String() string {
33 if s.Meta == "" {
34 return s.Status.String()
35 }
36 return fmt.Sprintf("%s(%s)", s.Status.String(), s.Meta)
37 }
38
39 // Map is a map of returned domains to their status
40 type Map map[string]Status
41
42 // Set adds the domain and status to the StatusMap
43 func (m Map) Set(domain string, status Status) {
44 m[domain] = status
45 }
46
47 // NewMap returns a new StatusMap containing the domain and status
48 func NewMap(domain string, status Status) Map {
49 m := make(Map)
50 m.Set(domain, status)
51 return m
52 }
53
54 // DomainStatus states
55 const (
56 UNKNOWN = iota
57 GOOD = iota
58 TIMEOUT = iota
59 NOHOST = iota
60 REFUSED = iota
61 ERROR = iota
62 REDIRECT = iota
63 CT = iota
64 )
65
66 // return domain status for printing
67 func (status DomainStatus) String() string {
68 switch status {
69 case UNKNOWN:
70 return "Unknown"
71 case GOOD:
72 return "Good"
73 case TIMEOUT:
74 return "Timeout"
75 case NOHOST:
76 return "No Host"
77 case REFUSED:
78 return "Refused"
79 case ERROR:
80 return "Error"
81 case REDIRECT:
82 return "Redirect"
83 case CT:
84 return "CT"
85 }
86 return "?"
87 }
88
89 // CheckNetErr check for errors, print if network related
90 func CheckNetErr(err error) DomainStatus {
91 if err == nil {
92 return GOOD
93 } else if netError, ok := err.(net.Error); ok && netError.Timeout() {
94 return TIMEOUT
95 } else {
96 switch t := err.(type) {
97 case *net.OpError:
98 if t.Op == "dial" {
99 return NOHOST
100 } else if t.Op == "read" {
101 return REFUSED
102 }
103 case syscall.Errno:
104 if t == syscall.ECONNREFUSED {
105 return REFUSED
106 }
107 }
108 }
109 return ERROR
110 }