pax_global_header00006660000000000000000000000064132055376230014520gustar00rootroot0000000000000052 comment=f67c0513a9ba84764fdae6afde5aa7fa4627ed63 WhatWeb-0.4.9/000077500000000000000000000000001320553762300130735ustar00rootroot00000000000000WhatWeb-0.4.9/.gitignore000066400000000000000000000000101320553762300150520ustar00rootroot00000000000000.projectWhatWeb-0.4.9/CHANGELOG000066400000000000000000000450011320553762300143050ustar00rootroot00000000000000 Version 0.4.9 November 23rd 2017 * Added unit testing with rake @bcoles * Added Elastic Search output @SlivTaMere * Source code formatting cleanup @Code0x58 * Thread reuse and logging through a single thread @Code0x58 * Fixed max-redirection bug @Code0x58 * Fixed bug when using a proxy and HTTPS (unknown user) * Fixed timeout deprecation warning @iGeek098 * New plugins and plugin updates @guikcd @bcoles @andreas-becker * Added proxy and user-agent to logging @rdubourguais * Updated Alexa top websites lists * Updated update-alexa script * Updated IP to Country database * Updated man page * Updated Mongo DB output for Mongo 2.x Version 0.4.8-dev (Continuous release 2012 - 2017) * Added support for all ciphers - TLSv1:TLSv1.1:TLSv1.2:SSLv3:SSLv2. Thanks @milo2012 * New colour scheme for brief output * New Verbose output * --color, --colour now takes case insensitive arguments, and can be enabled under Windows * HTTP Status descriptions are now included in brief and verbose output * Added short usage help * Converted all plugins to lowercase filenames * Updated plugins. Moved 100s of patterns in passive functions to matches[] * Added --search-plugins * Fixed bug with relative :url matching * Fixed bug with JSON output parsing * Fixed bug where :search => headers[xyz] was not case insensitive * Fixed bugs for Ruby 1.9.x and Ruby 2.x. Dropped support for Ruby 1.8.x. Thanks to nil0x42 and pvdl for bug fixes * Added TLSv1 support for Ruby 1.9.x and Ruby 2.x * Updated plugin list output * Updated plugin info output * Renamed scripts in plugin-development/ that update the Alexa lists and IP Country Database * Updated get-pattern for Ruby 2.x * Added over 700 new plugins * Added aggressive version detection using md5 static file matches to several plugins * Added support for raw HTTP headers when scanning local files * Added --dorks to return google dorks for the selected plugin * Added google dorks to more than 500 plugins * Added ./addons/hunter * Added ./addons/gggooglescan * Added ./addons/country-scanner * Added SQL logging with `--log-sql` and `--log-sql-create` arguments. * Added raw header support by monkey patching the net/http library * Added context searching for plugin matches[]. Added the matches keyword, :search. Values can be "headers","headers[server]"(or any other HTTP header),"body"(default), "all" (the raw headers + body) * Added methods for aggressive plugins to send HEAD and POST requests * Added --grep, -g option. Similar usage to --custom-plugin. (Requested by Scott Bell) * Removed the spidering feature and dependence on the customised and unsupported Anemone gem * Removed the extra_urls feature * Removed dependency on em-resolv-replace * Updated whatweb.xsl * Fixed a bug causing Mongo DB logging to fail * Fixed a bug causing brief logging to not escape special characters * Fixed meta refresh redirection but with HTML entities in the URL * Redesigned and refactored much of Whatweb's code. Introduced the Target class * Targets from input files are now executed ascending order * Better support for UTF-8 encoded strings in plugins. * :status and :url are now logical AND with other matches. They cannot match in isolation unless with each other. * Updated Country plugin. Fixed IPv6 bug * Changed version from 0.4.8 to 0.4.8-dev to show development version * Plugin brief output is now sorted alphabetically by plugin name * Removed plugin example URLs Version 0.4.7 Released April 5th 2011 * Performance enhancements & bug fixes * Added -p + as a shortcut for -p +plugins-disabled * Added --quiet, -q - to not display brief logging to STDOUT * Fix Makefile - you can now install whatweb over an old version * Removed certainty from Mongo and JSON output unless certainty < 100 * Removed certainty info from verbose output unless certainty <100 * Bugfixes for error reporting * Updated some error messages * Changed default open and read timeouts to 15 and 30 seconds respectively * Updated slow plugins * Added plugins: TVersity, Ultimate-Bulletin-Board, * Moved plugins to plugins-disabled: atom_feed, meta-city, meta-contact, meta-country, meta-geography, meta-state, meta-zipcode and script * Renamed mailto plugin to email Version 0.4.6 Released March 25th 2011 * Updated ~230 plugins * Added ~600 new plugins * Added Escenic CMS plugin from Erik Inge Bolsø * Added EscenicEngine5 plugin by nikosk * Added barracuda-load-balancer, binarysec-firewall, citrix-netscaler, cloudflare, evercookie, juniper-netscreen-secure-access, juniper-load-balancer, profense-firewall, vTigerCRM, watchguard-firewall, www-authenticate plugins by Aung Khant * Moved some plugins into disabled-plugins, as they clutter output. adobe_flash.rb, footer-hash.rb, frame.rb, header-hash.rb, md5.rb, script.rb, shortcut-icon.rb, tagpattern-hash.rb * Renamed disabled-plugins/ to plugins-disabled/ * Changed $ANEMONE_SKIP_REGEX=Regexp.union line to be compatible with Ruby 1.8.6. Thanks to Michal Ambroz * Added plugin reporting support for :model=>, :firmware=>, :module=> * Added --wait SECONDS between connections. Combine with -t 1 if preferred. * Added meta-refresh redirect support. eg. . Only for non-spidering * Added {:version=>/regexp/, :offset} to remove cargo cult programming. eg. {:version=>/2, :name=>"meta generator tag" } * Replaced :probability with :certainty in my-plugins/plugin-template.rb.txt. Thanks Erik Inge Bolsø * Added support for em-resolv-replace which speeds up whatweb many times. http://github.com/mperham/em-resolv-replace * Added XML stylesheet "whatweb.xsl" for XML reports * Added reporting of version detection with matches to the Plugin Info, eg. whatweb -I * Changed whatweb -I behaviour to search plugins for keywords. eg. './whatweb -I nuke' brings up ASP-Nuke, PHPNuke, DotNetNuke, etc. * Bugfix: Changed webpage data for when working with files, not URIs. Now it passes empty hashes, etc instead of nil which caused plugins to report errors. * Added MongoDB logging. Use with --log-mongo-database, --log-mongo-host, --log-mongo-collection, --log-mongo-username, --log-mongo-password. Only database has no default. * Added JSON logging. Must have the json ruby gem installed or be using Ruby 1.9 * Added MagicTree logging. * MagicTree logging updated by Gremwell. * Added error logging. * Added Verbose logging. * Added XML header and footer to XML logs * Modified XML logging to record modules separately * Bug fix: Escaping the XML log properly for &, <, >, " * All logs are now flushed/synced * Bug fix: References to :probability instead of :certainty in some logging * Changed error message for non resolving hostnames from "undefined method `closed?' for nil:NilClass" to "Cannot resolve hostname" * Added ascii whatweb logo * Moved Plugin class into lib/plugins.rb * Added startup and shutdown for plugins * Model and Firmware results now display in dark green * Added :filepath match type * Added vulnerability matching support, this is still in the experimental phase and not supported. * Added vulnerability matching code to the awstats plugin. * Precompiled regular expressions in matches[] for speed improvement * Changed internal sleep times from 1s to 0.5s * Added --debug to raise errors found in plugins * Usage displays faster when no arguments are provided * Added version string to the help usage * Added advanced plugin template * Removed How to write whatweb plugins text file as it's deprecated by the wiki * Brief output escapes [] and all characters before SPACE with URL encoding * Added --quiet, -q to suppress Brief Output on stdout by default. Thanks to cdybedahl for this idea. * Improved OSX compatibility with a patch from matti for symlinks * Added :status for HTTP Status codes to match[]. :status has a logical AND with a :url, it can't be by itself. * Updated plugin list and plugin info output * Bug fix: Now redirects for HTTP statuses 300 through 399. Previously redirected for 301,302 and 307. * Bug fix: :account didn't have regular expression support * Changed :modules to :module, deprecated :accounts to :account * Added redirect control. options are 'never',`http-only', `meta-only', `same-site', `same-domain', 'always' * Added --max-redirects. Control the maximum number of contiguous redirects followed * Added custom headers. Can be used multiple times. Examples: --header or -H. eg. "foo:bar" or "user-agent: blinky". Specifying a default header will replace it. Specifying an empty value removes hte header, eg. "User-Agent:" * Added support for HTTP basic authentication. -u and --user * Added plugin-development/get-pattern by Aung Khant * Added to plugin-development/: wget-alexa-top-1m, wget-ip-to-country, alexa-top-1000.txt, alexa-top-100.txt, wikipedia-top-1000.txt * Added nmap-style IP address range support Version 0.4.5 Released August 17th 2010 * Added 5 plugins from Tonmoy Saikia. They are: Commonspot, TextPattern, Mediawiki, DUclassified and Mailman * Added 119 plugins from Brendan Coles. They are: Alcatel-Lucent-Omniswitch, Allinta-CMS, anyInventory, Arab-Portal, AVTech-Video-Web-Server, Barracuda-Spam-Firewall, Basilic, Biromsoft-WebCam, BlueNet-Video-Server, BM-Classifieds, Brother-Printer, BusinessSpace, BXR, Campsite, Canon-Network-Camera, Cisco-VPN-3000-Concentrator, CMSQLite, ColdFusion, coWiki, cpCommerce, CruxCMS, CruxPA, Dell-Printer, D-Link-Network-Camera, DMXReady, DT-Centrepiece, EazyCMS, eLitius, EMO-Realty-Manager, Empire-CMS, envezion~media, eSyndiCat, Evo-Cam, FestOS, Flax-Article-Manager, FluentNET, Forest-Blog, GuppY, HP-LaserJet-Printer, i-Catcher-Console, iDVR, Intellinet-IP-Camera, Interspire-Shopping-Cart, IPCop-Firewall, IQeye-Netcam, iRealty, iScripts-CyberMatch, iScripts-EasySnaps, iScripts-MultiCart, iScripts-ReserveLogic, iScripts-SocialWare, JAMM-CMS, Jamroom, Linksys-NAS, Linksys-Network-Camera, Linksys-Wireless-G-Camera, LocazoList-Classifieds, Lucky-Tech-iGuard, Mobotix-Network-Camera, MyioSoft-Ajax-Portal, My-PHP-Indexer, My-WebCamXP-Server, NetBotz-Network-Monitoring-Device, Netious-CMS, Netsnap-Web-Camera, Nukedit, Open-Blog, ORCA-Platform, ORITE-301-Camera, PageUp-People, Panasonic-Network-Camera, Parked-Domain, PHPDirector, PHPEasyData, phPhotoAlbum, Pixel-Ads-Script, Pixie, Pligg-CMS, PortalApp, Pressflow, RunCMS, sabros.us, samPHPweb, SHOUTcast-Administrator, SimpNews, SkaLinks, SmodCMS, Snap-Appliance-Server, Softbiz-Freelancers-Script, Softbiz-Online-Auctions-Script, Softbiz-Online-Classifieds, Sony-Network-Camera, Sony-Video-Network-Station, Stardot-Express, StarDot-NetCam, Star-Network, Subdreamer-CMS, Subrion-CMS, SyndeoCMS, syntaxCMS, TaskFreak, Team-Board, The-PHP-Real-Estate-Script, TomatoCMS, Toshiba-Network-Camera, Veo-Observer, VisionGS-Webcam, WebDVR, WebEye-Network-Camera, WebPress, WhiteBoard, Winamp-Web-Interface, Windows-Internet-Printing, Xerox-Printers, xGB, XHP-CMS, Zeus-Cart, Zoph, Zyxel-Vantage-Service-Gateway * Added 11 plugins from Caleb Anderson. They are: AdobeFlash, AtomFeed, CodeIgniterProfiler, DublinCore, MicrosoftODBCError, MysqlSyntaxError, OpenGraphProtocol, OpenID, OpenSearch, PasswordField, RSSFeed * Updated plugins: Aardvark-Topsites-PHP, Confluence, Open-Source-Ticket-Request-System, PHP-Link-Directory, PHP-Shell, Vulnerable-to-XSS, Zoph * Updated mailto plugin * Verbose output now shows which patterns were matched within a plugin * Fixed bug: Removed Makefile reference to 'disabled-plugins' folder * Ruby 1.9 compatability fix. requires digest/md5 instead of md5 * Ruby 1.9 compatability fix. Replace UTF8 chars in frog-cms, dotnetnuke and mno-go-search and wordpress-supercache * Fixed spelling error of verion in help information * Fixed a typo where -t is shown as the command line option for proxies * Modified command line usage and is now in 80x24 terminal format * MD5sum of body is now available as @md5sum to all plugins * :md5 is available in matches[], eg. {:name=>"must be treshna.com",:md5=>"8666257030b94d3bdb46e05945f60b42"} * tag pattern of HTML elements in body is now available as @tagpattern to all plugins * :tagpattern is available in matches[], eg. {:name=>"must be google.com",:tagpattern=>""!doctype,html,head,meta,title,/title,script,/script,style,/style, etc...."} * :url is available in plugins. eg. {:url=>"/wp-login.php", :text=>'action=lostpassword'}, this will match the url and the text passively and when scanning aggressively, it will request the specified url and check for the text. Another example, {:url=>"/readme.html", :md5=>'9ea06ab0184049bf4ea2410bf51ce402', :version=>"3.0"}, * Added --url-prefix, eg. whatweb --url-prefix www.morningstarsecurity.com/ -i ./guess-files * Added --url-suffix, eg. whatweb --url-suffix /robots.txt -i ./target-urls * Added --url-pattern, eg. whatweb --url-pattern www.example.com/%insert%/.htaccess -i ./folder-list * Added --custom-plugin to define a plugin on the command line. eg, ./whatweb --custom-plugin ":text=>'powered by abc'" -i ./targets or --custom-plugin "{:text=>'powered by abc'},{:regexp=>/meta abc/i}" -i ./targets * Plugin errors are now in red, added target name * Added --open-timeout and --read-timeout * Removed div-span plugin, replaced with HTML tag pattern hash * Added --spider-skip-extensions. Redefine the file extensions that Anemone will skip. The list is comma delimited. * Moved plugin-template.rb to my-plugins and added more example, comments, etc * Added $DEBUG = false. If set to true, it will raise errors in plugins to assist plugin development. Version 0.4.4 Released June 29th 2010 * :probability is renamed to :certainty. :certainty in plugins is no longer required, it defaults to 100 if not specified. * Fixed bug with ruby 1.8.5 when loading plugins * Added author names to plugin info, eg. whatweb -I * Added 67 plugins from Brendan Coles, bringing WhatWeb up to 163 plugins. 360-Web-Manager,ANECMS,AWStats,Aardvark-Topsites-PHP,ArGoSoft-Mail-Server,Axis-Network-Camera,BeEF,BlognPlus,Burning-Board-Lite,CGI,CGIProxy,CMScontrol,CMSimple,Confluence,DUforum,DUgallery,F3Site,File-Upload-Manager,Google-API,Google-Hack-Honeypot,IMGallery,JGS-Portal,Kloxo,Liferay,Lime-Survey,Linksys-USB-HDD,Loggix,Microsoft-Sharepoint,Open-Freeway,Open-Source-Ticket-Request-System,PG-Roomate-Finder-Solution,PHP-Fusion,PHP-Layers,PHP-Link-Directory,PHP-Shell,PHPFM,PHPraid,PhilBoard,Piwik,QNAP-NAS,Saurus-CMS,Site-Sift,TWiki,Trac,Turbo-Seek,Umbraco,VideoShareEnterprise,Virtualmin,Vulnerable-To-XSS,WWWBoard,Web-Calendar-System,Web-Data-Administrator,WoW-Raid-Manager,X7-Chat,Zen-Cart,Zikula,boastMachine,ezBOO-WebStats,jobberBase,mojoPortal,php-ping,phpFreeChat,phpMyAdmin,phpPgAdmin,phpSysInfo,phpinfo,uPortal * Added references to Security-Assessment.com * Updates to README, CHANGELOG, plugin-template.rb.txt Version 0.4.3 Released May 24th 2010 * Added GPLv2 notices * Added Makefile (Thanks Michal Ambroz ) * Added man pages (Thanks Michal Ambroz ) * Added --version * Added Invalid command line argument handling * Added @cookie variable to plugins but is not availble for recursive use * Changed output colour of page titles * Changed plugin names to use a CamelCase convention * Merged the google analytics GA and Urchin plugins * Modified MovableType plugin * Added Cookie names plugin * Added Concrete5 CMS plugin * Added CushyCMS plugin * Added FrogCMS plugin * Added ModxCMS plugin * Added TypoLight plugin * Added ExpressionEngine plugin * Fixed a bug in Tomcat plugin * New feature, my-plugins/ folder. Keep your personal plugins separate. * Usage info shows correct defaults * Fixed a bug where aggressive plugins didn't use the proxy settings * Added XML (naive) logging * Updated usage to show how to pipe HTML to /dev/stdin * Added --no-redirect option. Do not follow HTTP 3xx redirects Version 0.4.2 Released April 30th 2010 * Added header-hash plugin. Makes a hash of the first 500 characters. This is useful to identify unknown systems * Added footer-hash plugin. Makes a hash of the last 500 characters, only if the page has > 1000 characters. This is useful to identify unknown systems * Added div-span-structure plugin. Makes a hash of a signature of div and span tags. This is useful to identify unknown systems * Added MikroTik Router plugin. Recognises version * Fixed a bug where the URL had a ? suffix. This caused some types of http servers to repspond incorrectly. * Added SquirrelMail plugin. Recognises version * Added SearchFitShoppingCart plugin. Recognises version * Added RoundCube plugin. * Modified OSCommerce plugin. Recognises security warnings about file permissions and installation directory. * Changed output colour to be more readable. Plugins that create hashes are in grey * Changed output order of plugins, so plugins that create hashes come last Version 0.4.1 Released April 28th 2010 * Removed dependency on rubygems and libxslt by modifying and locally including the Anemone gem. This also simplified installation * Fixed a bug which didn't send URL parameters. eg. would send /index.php instead of /index.php?q=foo * Improved installation instructions. Henri Salo contacted me to say ruby-dev is required for Anemone * Removed UTF-8 character in formmail * Changed require 'md5' to require 'digest/md5' for compatibility with ruby 1.9 * Fixed bug in Tomcat plugin * Added SilverStripe plugin * Added DotNetNuke plugin * Added HTML5 plugin * Added PHP error plugin * Modified PHP-Nuke plugin * Changed the plugin development script, wget-list to retry only twice * Added proxy support * Default threads is now 25 * Default max recursive spidering depth is now 10 * Default max number of links to follow on a single page is now 250 Version 0.4 Released March 13th 2010 * Added HTTPS support * Improved installation instructions * Improved documentation * Better compatibility with ruby 1.9. Changed a case statement syntax, changed when 0: to when 0 then. * Removed UTF-8 characters in plugins that were causing crashes * Added php-nuke plugin, passively recognises modules * Added Fluxbb plugin, can identify versions aggressively * Added meta powered-by plugin. Matches tags like * Added powered by plugin. Matches "Powered by BobsCMS", any text following powered by * Improved plugin info listing invoked by ./whatweb -I. Shows number of examples and matches, and shows presence of passive and aggressive functions * Changed output style. Before strings are surrounded by single quotes, now all strings are surrounded by square brackets * Added OpenCMS plugin submitted by Emilio Casbas * Added TomCat plugin submitted by Louis Nyffenegger * Improved meta-generator plugin * Fixed a bug in processing a target list from a file where a trailing space would be interpreted incorrectly Version 0.3 Released November 2nd 2009 at Kiwicon III WhatWeb-0.4.9/Gemfile000066400000000000000000000012041320553762300143630ustar00rootroot00000000000000## # This file is part of WhatWeb and may be subject to # redistribution and commercial restrictions. Please see the WhatWeb # web site for more information on licensing and terms of use. # http://www.morningstarsecurity.com/research/whatweb ## source 'https://rubygems.org' # JSON logging - optional group :json do #gem 'json' end # MongoDB logging - optional group :mongo do #gem 'mongo' #gem 'rchardet' end # Character set detection - optional group :rchardet do #gem 'rchardet' end # Development dependencies required for tests group :test do gem 'rake' gem 'minitest' gem 'rubocop' gem 'rdoc' gem 'bundler-audit' end WhatWeb-0.4.9/INSTALL000066400000000000000000000001541320553762300141240ustar00rootroot00000000000000INSTALL Visit https://github.com/urbanadventurer/WhatWeb/wiki/Installation for installation instructions. WhatWeb-0.4.9/LICENSE000066400000000000000000000432541320553762300141100ustar00rootroot00000000000000 GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. WhatWeb-0.4.9/Makefile000066400000000000000000000023641320553762300145400ustar00rootroot00000000000000NAME = whatweb PREFIX = /usr BINPATH = $(PREFIX)/bin LIBPATH = $(PREFIX)/share MANPATH = $(PREFIX)/share/man DOCPATH = $(PREFIX)/share/doc install: # upgrade/installation will leave the my-plugins folder rm -Rf $(DESTDIR)$(BINPATH)/$(NAME) $(DESTDIR)$(MANPATH)/man1/$(NAME).1 $(DESTDIR)$(MANPATH)/man1/$(NAME).1.gz $(DESTDIR)$(DOCPATH)/$(NAME) $(DESTDIR)$(LIBPATH)/$(NAME)/plugins-disabled $(DESTDIR)$(LIBPATH)/$(NAME)/plugins $(DESTDIR)$(LIBPATH)/$(NAME)/lib $(DESTDIR)$(LIBPATH)/$(NAME)/plugin-development $(DESTDIR)$(LIBPATH)/$(NAME)/addons install -p -D -m 755 $(NAME) $(DESTDIR)$(BINPATH)/$(NAME) install -p -D -m 644 $(NAME).1 $(DESTDIR)$(MANPATH)/man1/$(NAME).1 gzip -f $(DESTDIR)$(MANPATH)/man1/$(NAME).1 install -d $(DESTDIR)$(LIBPATH)/$(NAME) install -d $(DESTDIR)$(DOCPATH)/$(NAME) cp -p -r my-plugins $(DESTDIR)$(LIBPATH)/$(NAME)/ cp -p -r plugins-disabled plugins lib plugin-development addons $(DESTDIR)$(LIBPATH)/$(NAME)/ cp -p -r CHANGELOG INSTALL LICENSE README whatweb.xsl $(DESTDIR)$(DOCPATH)/$(NAME)/ clean: # clean will remove your my-plugins folder. be warned rm -Rf $(DESTDIR)$(BINPATH)/$(NAME) $(DESTDIR)$(MANPATH)/man1/$(NAME).1 $(DESTDIR)$(MANPATH)/man1/$(NAME).1.gz $(DESTDIR)$(LIBPATH)/$(NAME) $(DESTDIR)$(DOCPATH)/$(NAME) WhatWeb-0.4.9/README000066400000000000000000000521331320553762300137570ustar00rootroot00000000000000.$$$ $. .$$$ $. $$$$ $$. .$$$ $$$ .$$$$$$. .$$$$$$$$$$. $$$$ $$. .$$$$$$$. .$$$$$$. $ $$ $$$ $ $$ $$$ $ $$$$$$. $$$$$ $$$$$$ $ $$ $$$ $ $$ $$ $ $$$$$$. $ `$ $$$ $ `$ $$$ $ `$ $$$ $$' $ `$ `$$ $ `$ $$$ $ `$ $ `$ $$$' $. $ $$$ $. $$$$$$ $. $$$$$$ `$ $. $ :' $. $ $$$ $. $$$$ $. $$$$$. $::$ . $$$ $::$ $$$ $::$ $$$ $::$ $::$ . $$$ $::$ $::$ $$$$ $;;$ $$$ $$$ $;;$ $$$ $;;$ $$$ $;;$ $;;$ $$$ $$$ $;;$ $;;$ $$$$ $$$$$$ $$$$$ $$$$ $$$ $$$$ $$$ $$$$ $$$$$$ $$$$$ $$$$$$$$$ $$$$$$$$$' ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Readme for WhatWeb - Next generation web scanner. Developed by Andrew Horton aka urbanadventurer and Brendan Coles Version: 0.4.9. November 23rd, 2017 License: GPLv2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This product is subject to the terms detailed in the license agreement. For more information about WhatWeb visit: Homepage: http://www.morningstarsecurity.com/research/whatweb Wiki: https://github.com/urbanadventurer/WhatWeb/wiki/ If you have any questions, comments or concerns regarding WhatWeb, please consult the documentation prior to contacting one of the developers. Your feedback is always welcome. Contents ======================================================================== 1. About WhatWeb 2. Example Usage 3. Usage 4. Logging & Output 5. Plugins 6. Aggression 7. Performance & Stability 8. Optional Dependencies 9. Release History 10. Credits 11. Updates & Additional Information ======================================================================== 1. About WhatWeb ================================================================================ WhatWeb identifies websites. Its goal is to answer the question, "What is that Website?". WhatWeb recognises web technologies including content management systems (CMS), blogging platforms, statistic/analytics packages, JavaScript libraries, web servers, and embedded devices. WhatWeb has over 1700 plugins, each to recognise something different. WhatWeb also identifies version numbers, email addresses, account IDs, web framework modules, SQL errors, and more. WhatWeb can be stealthy and fast, or thorough but slow. WhatWeb supports an aggression level to control the trade off between speed and reliability. When you visit a website in your browser, the transaction includes many hints of what web technologies are powering that website. Sometimes a single webpage visit contains enough information to identify a website but when it does not, WhatWeb can interrogate the website further. The default level of aggression, called 'stealthy', is the fastest and requires only one HTTP request of a website. This is suitable for scanning public websites. More aggressive modes were developed for use in penetration tests. Most WhatWeb plugins are thorough and recognise a range of cues from subtle to obvious. For example, most WordPress websites can be identified by the meta HTML tag, e.g. '', but a minority of WordPress websites remove this identifying tag but this does not thwart WhatWeb. The WordPress WhatWeb plugin has over 15 tests, which include checking the favicon, default installation files, login pages, and checking for "/wp-content/" within relative links. Features: * Over 1700 plugins * Control the trade off between speed/stealth and reliability * Performance tuning. Control how many websites to scan concurrently. * Multiple log formats: Brief (greppable), Verbose (human readable), XML, JSON, MagicTree, RubyObject, MongoDB, ElasticSearch, SQL. * Proxy support including TOR * Custom HTTP headers * Basic HTTP authentication * Control over webpage redirection * Nmap-style IP ranges * Fuzzy matching * Result certainty awareness * Custom plugins defined on the command line 2. Example Usage ================================================================================ Using WhatWeb on a couple of websites (standard WhatWeb output is in colour): $ ./whatweb slashdot.org reddit.com http://reddit.com [302] HTTPServer[AkamaiGHost], RedirectLocation[http://www.reddit.com/], Via-Proxy[1.1 bc1], IP[173.223.232.64], Akamai-Global-Host, Country[UNITED STATES][US] http://slashdot.org [200] Script, HTTPServer[Unix][Apache/1.3.42 (Unix) mod_perl/1.31], Google-Analytics[GA][32013], Via-Proxy[1.1 bc5], UncommonHeaders[x-fry,x-varnish,x-xrds-location,slash_log_data], Apache[1.3.42][mod_perl/1.31], HTML5, IP[216.34.181.45], OpenGraphProtocol[100000696822412], X-Powered-By[Slash 2.005001], Title[Slashdot: News for nerds, stuff that matters], Email[canadaboy@nOspam.gmail.com,jbort@nww.com], Country[UNITED STATES][US] http://www.reddit.com/ [200] Frame, PasswordField[passwd,passwd2], Script, HTTPServer['; DROP TABLE servertypes; --], IP[203.97.86.202], JQuery, Cookies[reddit_first], Title[reddit: the voice of the internet -- news before it happens], Country[NEW ZEALAND][NZ] 3. Usage ================================================================================ .$$$ $. .$$$ $. $$$$ $$. .$$$ $$$ .$$$$$$. .$$$$$$$$$$. $$$$ $$. .$$$$$$$. .$$$$$$. $ $$ $$$ $ $$ $$$ $ $$$$$$. $$$$$ $$$$$$ $ $$ $$$ $ $$ $$ $ $$$$$$. $ `$ $$$ $ `$ $$$ $ `$ $$$ $$' $ `$ `$$ $ `$ $$$ $ `$ $ `$ $$$' $. $ $$$ $. $$$$$$ $. $$$$$$ `$ $. $ :' $. $ $$$ $. $$$$ $. $$$$$. $::$ . $$$ $::$ $$$ $::$ $$$ $::$ $::$ . $$$ $::$ $::$ $$$$ $;;$ $$$ $$$ $;;$ $$$ $;;$ $$$ $;;$ $;;$ $$$ $$$ $;;$ $;;$ $$$$ $$$$$$ $$$$$ $$$$ $$$ $$$$ $$$ $$$$ $$$$$$ $$$$$ $$$$$$$$$ $$$$$$$$$' WhatWeb - Next generation web scanner version 0.4.9. Developed by Andrew Horton aka urbanadventurer and Brendan Coles. Homepage: http://www.morningstarsecurity.com/research/whatweb Usage: whatweb [options] TARGET SELECTION: Enter URLs, hostnames, IP adddresses, filenames, or nmap-format IP address ranges. --input-file=FILE, -i Read targets from a file. You can pipe hostnames or URLs directly with -i /dev/stdin. TARGET MODIFICATION: --url-prefix Add a prefix to target URLs. --url-suffix Add a suffix to target URLs. --url-pattern Insert the targets into a URL. Requires --input-file, eg. www.example.com/%insert%/robots.txt AGGRESSION: The aggression level controls the trade-off between speed/stealth and reliability. --aggression, -a=LEVEL Set the aggression level. Default: 1. Aggression levels are: 1. Stealthy Makes one HTTP request per target. Also follows redirects. 3. Aggressive If a level 1 plugin is matched, additional requests will be made. 4. Heavy Makes a lot of HTTP requests per target. Aggressive tests from all plugins are used for all URLs. HTTP OPTIONS: --user-agent, -U=AGENT Identify as AGENT instead of WhatWeb/0.4.9. --header, -H Add an HTTP header. eg "Foo:Bar". Specifying a default header will replace it. Specifying an empty value, eg. "User-Agent:" will remove the header. --follow-redirect=WHEN Control when to follow redirects. WHEN may be `never', `http-only', `meta-only', `same-site', `same-domain' or `always'. Default: always. --max-redirects=NUM Maximum number of contiguous redirects. Default: 10. AUTHENTICATION: --user, -u= HTTP basic authentication. --cookie, -c=COOKIES Provide cookies, e.g. 'name=value; name2=value2'. PROXY: --proxy Set proxy hostname and port. Default: 8080. --proxy-user Set proxy user and password. PLUGINS: --list-plugins, -l List all plugins. --info-plugins, -I=[SEARCH] List all plugins with detailed information. Optionally search with keywords in a comma delimited list. --search-plugins=STRING Search plugins for a keyword. --plugins, -p=LIST Select plugins. LIST is a comma delimited set of selected plugins. Default is all. Each element can be a directory, file or plugin name and can optionally have a modifier, eg. + or - Examples: +/tmp/moo.rb,+/tmp/foo.rb title,md5,+./plugins-disabled/ ./plugins-disabled,-md5 -p + is a shortcut for -p +plugins-disabled. --grep, -g=STRING Search for STRING in HTTP responses. Reports with a plugin named Grep. --custom-plugin=DEFINITION Define a custom plugin named Custom-Plugin, Examples: ":text=>'powered by abc'" ":version=>/powered[ ]?by ab[0-9]/" ":ghdb=>'intitle:abc \"powered by abc\"'" ":md5=>'8666257030b94d3bdb46e05945f60b42'" --dorks=PLUGIN List Google dorks for the selected plugin. OUTPUT: --verbose, -v Verbose output includes plugin descriptions. Use twice for debugging. --colour,--color=WHEN control whether colour is used. WHEN may be `never', `always', or `auto'. --quiet, -q Do not display brief logging to STDOUT. --no-errors Suppress error messages. LOGGING: --log-brief=FILE Log brief, one-line output. --log-verbose=FILE Log verbose output. --log-errors=FILE Log errors. --log-xml=FILE Log XML format. --log-json=FILE Log JSON format. --log-sql=FILE Log SQL INSERT statements. --log-sql-create=FILE Create SQL database tables. --log-json-verbose=FILE Log JSON Verbose format. --log-magictree=FILE Log MagicTree XML format. --log-object=FILE Log Ruby object inspection format. --log-mongo-database Name of the MongoDB database. --log-mongo-collection Name of the MongoDB collection. Default: whatweb. --log-mongo-host MongoDB hostname or IP address. Default: 0.0.0.0. --log-mongo-username MongoDB username. Default: nil. --log-mongo-password MongoDB password. Default: nil. --log-elastic-index Name of the index to store results. Default: whatweb --log-elastic-host Host:port of the elastic http interface. Default: 127.0.0.1:9200 PERFORMANCE & STABILITY: --max-threads, -t Number of simultaneous threads. Default: 25. --open-timeout Time in seconds. Default: 15. --read-timeout Time in seconds. Default: 30. --wait=SECONDS Wait SECONDS between connections. This is useful when using a single thread. HELP & MISCELLANEOUS: --short-help Short usage help. --help, -h Complete usage help. --debug Raise errors in plugins. --version Display version information. (WhatWeb 0.4.9). EXAMPLE USAGE: * Scan example.com. ./whatweb example.com * Scan reddit.com slashdot.org with verbose plugin descriptions. ./whatweb -v reddit.com slashdot.org * An aggressive scan of wired.com detects the exact version of WordPress. ./whatweb -a 3 www.wired.com * Scan the local network quickly and suppress errors. whatweb --no-errors 192.168.0.0/24 * Scan the local network for https websites. whatweb --no-errors --url-prefix https:// 192.168.0.0/24 * Scan for crossdomain policies in the Alexa Top 1000. ./whatweb -i plugin-development/alexa-top-100.txt \ --url-suffix /crossdomain.xml -p crossdomain_xml OPTIONAL DEPENDENCIES -------------------------------------------------------------------------------- To enable MongoDB logging install the mongo gem. To enable character set detection and MongoDB logging install the rchardet gem. 4. Logging & Output ================================================================================ The following types of logging are supported: --log-brief=FILE Brief, one-line, greppable format --log-verbose=FILE Verbose --log-xml=FILE XML format. XSL stylesheet is provided --log-json=FILE JSON format --log-json-verbose=FILE JSON verbose format --log-magictree=FILE MagicTree XML format --log-object=FILE Ruby object inspection format --log-mongo-database Name of the MongoDB database --log-mongo-collection Name of the MongoDB collection. Default: whatweb --log-mongo-host MongoDB hostname or IP address. Default: 0.0.0.0 --log-mongo-username MongoDB username. Default: nil --log-mongo-password MongoDB password. Default: nil --log-elastic-index Name of the index to store results. Default: whatweb --log-elastic-host Host:port of the elastic http interface. Default: 127.0.0.1:9200 --log-errors=FILE Log errors. This is usually printed to the screen in red. You can output to multiple logs simultaneously by specifying multiple command line logging options. Advanced users who want SQL output should read the source code to see unsupported features. 5. Plugins ================================================================================ Matches are made with: * Text strings (case sensitive) * Regular expressions * Google Hack Database queries (limited set of keywords) * MD5 hashes * URL recognition * HTML tag patterns * Custom ruby code for passive and aggressive operations To list the plugins supported: $ ./whatweb -l WhatWeb Plugin List Plugin Name - Description -------------------------------------------------------------------------------- 1024-CMS - 1024 is one of a few CMS's leading the way with the implementation... 360-Web-Manager - 360-Web-Manager 3COM-NBX - 3COM NBX phone system. The NBX NetSet utility is a web interface i... 3dcart - 3dcart - The 3dcart Shopping Cart Software is a complete ecommerce s... 4D - 4D web application deployment server 4images - 4images is a powerful web-based image gallery management system. Fe... ... (truncated) To view more detail about a plugin or search plugins for a keyword: $ ./whatweb -I phpBB WhatWeb Detailed Plugin List Searching for phpBB ================================================================================ Plugin: phpBB -------------------------------------------------------------------------------- Description: phpBB is a free forum Website: http://phpbb.org/ Author: Andrew Horton Version: 0.3 Features: [Yes] Pattern Matching (7) [Yes] Version detection from pattern matching [Yes] Function for passive matches [Yes] Function for aggressive matches [Yes] Google Dorks (1) Google Dorks: [1] "Powered by phpBB" ================================================================================ All plugins are loaded by default. Plugins can be selected by directories, files or plugin names as a comma delimited list with the -p or --plugin command line option. Each list item may have a modifier: + adds to the full set, - removes from the full set and no modifier overrides the defaults. Examples : --plugins +plugins-disabled,-foobar --plugins +/tmp/moo.rb --plugins foobar (only select foobar) -p title,md5,+./plugins-disabled/ -p ./plugins-disabled,-md5 The --dorks command line option returns google dorks for the selected plugin. For example, --dorks wordpress returns "is proudly powered by WordPress" The --grep, -g command line option searches the target page for the selected string and returns a match in a plugin called Grep if it is found. 6. Aggression ================================================================================ WhatWeb features several levels of aggression. By default the aggression level is set to 1 (stealthy) which sends a single HTTP GET request and also follows redirects. --aggression, -a 1. Stealthy Makes one HTTP request per target. Also follows redirects. 2. Unused 3. Aggressive Can make a handful of HTTP requests per target. This triggers aggressive plugins for targets only when those plugins are identified with a level 1 request first. 4. Heavy Makes a lot of HTTP requests per target. Aggressive tests from all plugins are used for all URLs. Level 3 aggressive plugins will guess more URLs and perform actions that are potentially unsuitable without permission. WhatWeb currently does not support any intrusion/exploit level tests in plugins. An example of the different results between level 1 and level 3: ----------------------------------------------------------------- A level 1, stealthy scan identifes that smartor.is-root.com/forum/ uses phpBB version 2: $ ./whatweb smartor.is-root.com/forum/ http://smartor.is-root.com/forum/ [200] PasswordField[password], HTTPServer[Apache/2.2.15], PoweredBy[phpBB], Apache[2.2.15], IP[88.198.177.36], phpBB[2], PHP[5.2.13], X-Powered-By[PHP/5.2.13], Cookies[phpbb2mysql_data,phpbb2mysql_sid], Title[Smartors Mods Forums - Reloaded], Country[GERMANY][DE] A level 3, aggressive scan triggers additional tests in the phpBB plugin which identifies that the website uses phpBB version 2.0.20 or higher: $ ./whatweb -p plugins/phpbb.rb -a 3 smartor.is-root.com/forum/ http://smartor.is-root.com/forum/ [200] phpBB[2,>2.0.20] Note the use of the -p argument to select only the phpBB plugin. It is advisable, but not mandatory, to select a specific plugin when attempting to fingerprint software versions in aggressive mode. This approach is far more stealthy as it will limit the number of requests. WhatWeb has no caching so if you use aggressive plugins on redirecting URLs you may fetch the same files multiple times. 7. Performance & Stability ================================================================================ WhatWeb features several options to increase performance and stability. --max-threads, -t Number of simultaneous threads. Default: 25. --open-timeout Time in seconds. Default: 15 --read-timeout Time in seconds. Default: 30 --wait=SECONDS Wait SECONDS between connections This is useful when using a single thread. The --wait and --max-threads commands can be used to assist in IDS evasion. Changing the user-agent using the -U or --user-agent command line option will avoid the Snort IDS rule for WhatWeb. If you are scanning ranges of IP addresses, it is much more efficient to use a port scanner like nmap to discover which have port 80 open before scanning with WhatWeb. Character set detection, with the Charset plugin dramatically decreases performance by requiring more CPU. This is required by JSON and MongoDB logging. 8. Optional Dependencies ================================================================================ To enable JSON logging install the json gem. gem install json To enable MongoDB logging install the mongo gem. gem install mongo To enable character set detection and MongoDB logging install the rchardet gem. gem install rchardet cp plugins-disabled/charset.rb my-plugins/ 9. Release History ================================================================================ Version 0.3 Released at Kiwicon III (kiwicon.org), November 2nd, 2009 Version 0.4 Released March 14th, 2010 Version 0.4.1 Released April 28th, 2010 Version 0.4.2 Released April 30th, 2010 Version 0.4.3 Released May 24th, 2010 Version 0.4.4 Released June 29th, 2010 Version 0.4.5 Released August 17th, 2010 Version 0.4.6 Released March 25th, 2011 Version 0.4.7 Released April 5th, 2011 Version 0.4.8-dev (Continuous release from 2012 to 2017) Version 0.4.9 Released November 23rd, 2017 10. Credits ================================================================================ Written by urbanadventurer aka Andrew Horton and Brendan Coles Homepage: http://www.morningstarsecurity.com/research/whatweb License: GPLv2 DEVELOPERS Andrew Horton Brendan Coles CONTRIBUTORS Thank you to the following people who have contributed to WhatWeb. Emilio Casbas Louis Nyffenegger Patrik Wallström (pawal) Caleb Anderson (dirtyfilthy) Tonmoy Saikia Aung Khant (yehgdotnet) Erik Inge Bolsø nk@dsigned.gr Steve Milner (ashcrow) Michal Ambroz Gremwell Sagar Prakash Junnarkar (sagarjunnarkar) GertBerger Quintin Poirier Eric Sesterhenn dengjw (jawa) Pedro Worcel (droop) Matthieu Keller (maggick) Peter (pvdl) Napz (RootCon) nilx042 Fabian Affolter (fabaff) Andrew Silvernail (buff3r) Andre Ricardo (andrericardo) nikosk Patrick Thomas (coffeetocode) Guillaume Delcaour (guikcd) Sean (wiifm69) Matthieu Keller (maggick) Raul (raurodse) Andrew Petro (apetro) Artem Taranyuk (610) Matti Paksula (matti) Tim Smith (tas50) Sarthak Munshi (saru95) @rdubourguais @SlivTaMere @Code0x58 @iGeek098 @andreas-becker Please let me know if I need to add any more names. 11. Updates & Additional Information ================================================================================ The WhatWeb development build features regular updates. * WhatWeb-dev: https://github.com/urbanadventurer/WhatWeb/ * WhatWeb-dev-unstable: https://github.com/bcoles/WhatWeb/ Browse the wiki for more documentation and advanced usage techniques. * Wiki: https://github.com/urbanadventurer/WhatWeb/wiki/ WhatWeb-0.4.9/Rakefile000066400000000000000000000032271320553762300145440ustar00rootroot00000000000000## # This file is part of WhatWeb and may be subject to # redistribution and commercial restrictions. Please see the WhatWeb # web site for more information on licensing and terms of use. # http://www.morningstarsecurity.com/research/whatweb ## $DEBUG = false $VERBOSE = false require 'rake/testtask' require 'rubocop/rake_task' desc 'Run all tests' task :all do puts 'Running unit tests' Rake::Task['unit'].invoke end task :default => :unit Rake::TestTask.new(:unit) do |t| t.description = 'Run unit tests' t.test_files = FileList['test/unit.rb'] end desc 'Run bundle-audit' task :bundle_audit do Rake::Task['bundle_audit:update'].invoke Rake::Task['bundle_audit:check'].invoke end desc 'Generate API documentation to doc/rdocs/index.html' task :rdoc do Rake::Task['rdoc:rerdoc'].invoke end RuboCop::RakeTask.new ############################################################ # rdoc ############################################################ namespace :rdoc do require 'rdoc/task' desc 'Generate API documentation to doc/rdocs/index.html' Rake::RDocTask.new do |rd| rd.rdoc_dir = 'doc/rdocs' rd.main = 'README' rd.rdoc_files.include( 'whatweb', 'lib/*\.rb') rd.options << '--line-numbers' rd.options << '--all' end end ############################################################ # bundle-audit ############################################################ namespace :bundle_audit do require 'bundler/audit/cli' desc 'Update bundle-audit database' task :update do Bundler::Audit::CLI.new.update end desc 'Check gems for vulns using bundle-audit' task :check do Bundler::Audit::CLI.new.check end end WhatWeb-0.4.9/addons/000077500000000000000000000000001320553762300143435ustar00rootroot00000000000000WhatWeb-0.4.9/addons/country-scanner000077500000000000000000000065371320553762300174360ustar00rootroot00000000000000#!/bin/bash # country scanner v1.1 # Illustrative example of how to write scripts using whatweb and nmap. # Automatically discover samples of web servers and test whatweb # GRAY="\033[1;30m" RED="\033[0;31m" LIGHT_RED="\033[1;31m" GREEN="\033[0;32m" LIGHT_GRAY="\033[0;37m" CYAN="\033[0;36m" LIGHT_CYAN="\033[1;36m" DARK_GREY="\033[1;30m" BROWN="\033[0;33m" WHITE="\033[1;37m" BLUE="\033[0;34m" LIGHT_BLUE="\033[1;34m" NO_COLOUR="\033[0m" SUBTITLE=$DARK_GREY echo -e " $LIGHT_RED __ ____ ____ __ __ _____/ |________ ___.__.$RED _/ ___\ / \| | \/ \ __\_ __ \\\\\ | | \ \___( // ) | / | \ | | | \/ \___ | \_____>\____/|____/|___| /__| |__| / ____| \/ \/ $LIGHT_BLUE ______ ____ _____ ____ ____ ____ _______ $BLUE / ___/_/ ___\ \__ \ / \ / \_/ __ \\\\\_ __ \\ \___ \ \ \___ / \\\\\ \_| | \ | \ ___/ | | \/ /______> \___ >(____ /|___| /___| /\_____>|__| \/ \/ \/ \/ $NO_COLOUR Country Scanner - Sample the web around the world Version v1.1 by urbanadventurer " N=1000 SCANBLOCK=10000 AGENT="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; eSobiSubscriber 2.0.4.16; .NET CLR 3.0.30618)" usage(){ echo "Usage: $0 " echo -e "-c\tCountry" echo -e "-l\tList countries" echo -e "-h\tHelp" echo -e "-n\tNumber of whatweb log entries. Default: $N" echo } checkdependencies(){ for c in `echo "nmap geoipgen"`; do if [ -z "`which $c`" ]; then echo "$c not found. Aborting" echo "You need to install geoipgen to generate country IP lists" echo "Visit http://www.morningstarsecurity.com/research/geoipgen" exit 1 fi done } checkdependencies while getopts "hln:c:" OPTION do case $OPTION in h) usage;exit 1;; n) N="$OPTARG";; l) geoipgen -l;exit 1;; c) CC="$OPTARG";; ?) usage; exit ;; esac done if [ -z "$CC" ]; then usage; exit 1 fi # check CC is sane # find whatweb WHATWEB=`dirname "$0"`"/../whatweb" FOUND=0 f=`tempfile -d . --prefix scan- --suffix -geo` g=`tempfile -d . --prefix scan- --suffix -nmp` h=`tempfile -d . --prefix scan- --suffix "-$CC"` echo "Scan of $CC started at `date`" echo "--------------------------------------------------" echo while [ "$FOUND" -le "$N" ]; do echo "[*] Generating $SCANBLOCK IP addresses in $CC" echo " geoipgen -n \"$SCANBLOCK\" \"$CC\" > \"$f\"" geoipgen -n "$SCANBLOCK" "$CC" > "$f" echo echo "[*] Port scanning for web servers" echo " nmap --open -PN -n -p 80 -i \"$f\" -oG \"$g\" --max-retries 1 --max-rtt-timeout 30s --min-hostgroup 4096 --host-timeout 30s > /dev/null 2>/dev/null" nmap --open -PN -n -p 80 -i "$f" -oG "$g" --max-rtt-timeout 10s --min-hostgroup 4096 --host-timeout 10s > /dev/null 2>/dev/null echo fgrep open "$g" | cut -d ' ' -f 2 > "$f" #rm -f "$g" echo "[*] Found "`wc -l "$f" |cut -d ' ' -f 1`" IPs with TCP port 80 open" echo echo "[*] Scanning with WhatWeb. Logging to $h" # -p + echo " $WHATWEB --no-errors -U \"$AGENT\" -t 500 --read-timeout 30 --log-brief \"$h\" -i \"$f\"" $WHATWEB --no-errors -U "$AGENT" -t 50 --log-brief "$h" -i "$f" echo FOUND=`wc -l "$h" | cut -d ' ' -f 1` #rm -f "$f" echo "[*] Found $FOUND web servers so far" echo done echo "[*] Finished at `date`" rm -f "$f" "$g" WhatWeb-0.4.9/addons/gggooglescan000077500000000000000000000452361320553762300167420ustar00rootroot00000000000000#!/bin/bash # gggooglescan - Enumerate hostnames and URLs from Google. # Features: antibot avoidance, search within a country, custom search appliance, horizontal searching # # By Andrew Horton aka urbanadventurer, Security-Assessment.com # Homepage: www.morningstarsecurity.com/research/ # License: GPLv3 # Version o.5, o7/2011 # # Usage Notes: # GOOGLE=210.55.180.157 <-- this IP is within New Zealand # Be careful not to get banned # function list_useragents { for ((i=0; i<${#AGENTS[@]}; i+=1 )) { # do echo [$i] ${AGENTS[i]}; } } function set_agent() { if [[ "$1" =~ ^[0-9]+$ ]] ; then AGENT=${AGENTS[$1]} else AGENT="$1" fi } AGENTS=( "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; eSobiSubscriber 2.0.4.16; .NET CLR 3.0.30618)" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 1.1.4322)" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.11) Gecko/20101012 Firefox/3.6.11" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.6)" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618)" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.127 Safari/533.4" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 1.1.4322; InfoPath.2; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30618; .NET4.0C)" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_1; en-us) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.1.249.1064 Safari/532.5" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.8) Gecko/20100723 Ubuntu/10.04 (lucid) Firefox/3.6.8 GTB7.1" "Mozilla/5.0 (Windows NT 5.1; rv:2.0) Gecko/20100101 Firefox/4.0" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.99 Safari/533.4" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-us) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 (.NET CLR 3.5.30729)" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101026 SUSE/3.6.12-0.6.1 Firefox/3.6.12" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; WOW64; SLCC1; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729)" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1)" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; InfoPath.2)" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2) Gecko/20100115 Firefox/3.6 (.NET CLR 3.5.30729)" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.195.38 Safari/532.0" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.70 Safari/533.4" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.0.04506; .NET CLR 1.1.4322; Tablet PC 2.0)" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.04 (lucid) Firefox/3.6.13 GTB7.1" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10 ( .NET CLR 3.5.30729)" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C)" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.195.27 Safari/532.0" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.215 Safari/534.10" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.19) Gecko/2010031422 Firefox/3.0.19" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_4_11; en) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/4.1.3 Safari/533.19.4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB6.6; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; HPDTDF; .NET4.0C)" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 ( .NET CLR 3.5.30729)" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729)" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.44 Safari/534.7" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 (.NET CLR 3.5.30729)" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3 (.NET CLR 3.5.30729)" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.99 Safari/533.4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322)" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.44 Safari/534.7" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-us) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.7) Gecko/20100713 Firefox/3.6.7" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5" "Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/532.3 (KHTML, like Gecko) Chrome/4.0.223.11 Safari/532.3" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0; InfoPath.3)" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16" "Opera/9.80 (X11; Linux i686; U; en) Presto/2.2.15 Version/10.00" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-us) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322)" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12" "Mozilla/4.0 (compatible; MSIE 7.0;Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727;.NET CLR 3.0.04506.30)" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2)" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB0.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; HPDTDF)" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_4_11; en) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/4.1.2 Safari/533.18.5" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.16) Gecko/20101130 Firefox/3.5.16 ( .NET CLR 3.5.30729)" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) Gecko/20100915 Ubuntu/10.04 (lucid) Firefox/3.6.10 GTB7.1" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6; InfoPath.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729)" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.15) Gecko/2009102814 Ubuntu/8.10 (intrepid) Firefox/3.0.15" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729)" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.1.4) Gecko/20091016 Firefox/3.5.4 (.NET CLR 3.5.30729)" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.04 (lucid) Firefox/3.6.12 GTB7.1" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)" ) VERSION="o.5" GOOGLE=www.google.com COMMENTS=1 HOSTNAMESONLY=0 LOGFILE= DEPTH=5 AGENT=${AGENTS[ $(($RANDOM % ${#AGENTS[@]})) ]} # random user agent NUMRESULTS=10 DICTIONARY= WAIT=0 BOT_SLEEP=60m PASSCURL= INPUTFILE= while getopts 'c:d:n:g:l:i:u:e:s:x:otq' OPTION do case $OPTION in c) CC=`echo $OPTARG | tr '[:lower:]' '[:upper:]' `;; d) DEPTH=$OPTARG ;; # n) NUMRESULTS=$OPTARG ;; g) GOOGLE=$OPTARG ;; l) LOGFILE=$OPTARG ;; o) HOSTNAMESONLY=1 ;; i) INPUTFILE=$OPTARG ;; u) set_agent $OPTARG ;; e) DICTIONARY=$OPTARG ;; s) WAIT=$OPTARG ;; q) COMMENTS=0 ;; x) PASSCURL=$OPTARG ;; t) list_useragents;exit ;; esac done shift $(($OPTIND -1 )) if [ -z "$1" ] && [ -z "$INPUTFILE" ] ; then GRAY="\033[1;30m" RED="\033[0;31m" GREEN="\033[0;32m" LIGHT_GRAY="\033[0;37m" CYAN="\033[0;36m" LIGHT_CYAN="\033[1;36m" DARK_GREY="\033[1;30m" BROWN="\033[0;33m" WHITE="\033[1;37m" BLUE="\033[0;34m" LIGHT_BLUE="\033[1;34m" NO_COLOUR="\033[0m" SUBTITLE=$DARK_GREY echo -e "$LIGHT_GRAY _______ _____ _____ __ $LIGHT_GRAY| __| __| __|.-----..-----..-----.| |.-----.$LIGHT_GRAY.-----..----..---.-..-----. $LIGHT_BLUE| | | | | | || o || o || o || || o__|$LIGHT_BLUE|__ --|| __|| o || | $BLUE|_______|_____|_____||_____||_____||___ ||__||_____|$BLUE|_____||____||___._||__|__| $SUBTITLE G-G-Googlescan v$VERSION (o7/2o1o) $BLUE |_____| $SUBTITLE by urbanadventurer$NO_COLOUR ------------.------------------------------------------------------------------- Description | Enumerate hostnames and URLs using the Google Search Engine Homepage | www.morningstarsecurity.com/research/gggooglescan Author | Andrew Horton (urbanadventurer) from Security-Assessment.com Features | Antibot avoidance, Search within a country, Horizontal searching, | URL encoding and more ------------'------------------------------------------------------------------- Usage: $0 [OPTION]... [QUERY|-i] -i=INPUTFILE\tFile containing search queries -e=WORDLIST\tCombine each word from a wordlist with the query (avoid deep \t\tqueries), can combine with -i -c=CC\t\tSearch within a country, eg. au, uk or nz -d=NUM\t\tDepth of results. Num of pages to return. Default: $DEPTH -g=IP\t\tIP or hostname of a Google search website. \t\tDefault: $GOOGLE -l=FILE\tLog file, output is appended if the file already exists -o\t\tOutput hostnames, instead of URLs -u=AGENT|NUM\tUser Agent. Default is to randomly select one -t\t\tList user agents -s=SECONDS\tSleep for SECONDS between each query. Default: $WAIT -q\t\tQuiet. Turn off comment lines. -x\t\tPass cmdline args to curl, eg. -x \"--socks5 127.0.0.1:8118\" " # -n=NUM\t\tNumber of results per page. Default: $NUMRESULTS exit fi #echo "# User-Agent: $AGENT" function run_query { query="$1" if (( $COMMENTS )); then echo "# Query: $query" fi TMPFILE=`mktemp temp-gggXXXXXXXXXX` || exit 1 # escape query string # Thanks blueyed for the perl snippet at http://stackoverflow.com/questions/296536/urlencode-from-a-bash-script query=`echo -n "$query" | perl -p -e 's/([^A-Za-z0-9])/sprintf("%%%02X", ord($1))/seg'` if [ -n "$CC" ]; then country="&meta=cr%3Dcountry$CC" else country="" fi for (( i=0; $i<$DEPTH; i++ )) do if [ i = 0 ]; then start="" else start="&start=$(( i * 10 ))" fi URL="http://$GOOGLE/search?q=$query$country$start&num=$NUMRESULTS" if (( $COMMENTS )); then echo "# $URL" >&2 fi curl "$PASSCURL" -A "$AGENT" "$URL" 2>/dev/null > $TMPFILE LINKS=`cat $TMPFILE | egrep -o "href=\"http[^\"]+" | cut -d '"' -f 2` # remove links in the blacklist LINKS=`echo "$LINKS" | egrep -v "(www.youtube.com)|(google.com)|(/search\?q=cache)"` if (( $HOSTNAMESONLY )); then echo "$LINKS" | cut -d / -f 3 | cut -d : -f 1 | grep -v "^$" | tee -a $LOGFILE else echo "$LINKS" | grep -v "^$" | tee -a $LOGFILE fi # test for end of results if `grep -q "In order to show you the most relevant results, we have omitted some entries very similar" $TMPFILE`; then if (( $COMMENTS )); then echo "# results omitted page" >&2 fi rm -f $TMPFILE break fi if `grep -q "

Bad Request

" $TMPFILE`; then if (( $COMMENTS )); then echo "# Bad Request. You've done something wrong :|" >&2 fi rm -f $TMPFILE break fi if `grep -q "http://sorry.google.com/sorry/?continue=" $TMPFILE`; then if (( $COMMENTS )); then echo "# You're acting like a bot. Waiting for $BOT_SLEEP" >&2 fi sleep $BOT_SLEEP rm -f $TMPFILE break fi if ! `grep -q "&start=$(( (i+1) * 10 ))" $TMPFILE`; then if (( $COMMENTS )); then echo "# next page of results link missing" >&2 fi rm -f $TMPFILE break fi rm -f $TMPFILE sleep $WAIT done } if [ -n "$INPUTFILE" ]; then cat "$INPUTFILE" | while read QUERY; do if [ -n "$DICTIONARY" ]; then while read line; do echo "# $QUERY $line" run_query "$QUERY $line" done < "$DICTIONARY" else run_query "$QUERY" fi done else QUERY="$1" if [ -n "$DICTIONARY" ]; then while read line; do echo "# $QUERY $line" run_query "$QUERY $line" done < "$DICTIONARY" else run_query "$QUERY" fi fi WhatWeb-0.4.9/addons/hunter000077500000000000000000000123141320553762300155770ustar00rootroot00000000000000#!/bin/bash # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # This file is part of WhatWeb and may be subject to # redistribution and commercial restrictions. Please see the WhatWeb # web site for more information on licensing and terms of use. # http://www.morningstarsecurity.com/research/whatweb # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # Hunter finds web applications with Google then fingerprints them with WhatWeb # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # VERSION="0.1.2" AUTHOR="Brendan Coles [ itsecuritysolutions.org ]" # Hunter settings VERBOSE="TRUE" LIST="" QUERY="" APP="" LOG="hunter.$(date +"%Y%m%d%H%M%S")-$$.log" # WhatWeb settings AGGRESSION="1" USERAGENT="Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341" PROXYUSER="" PROXY="" FILTER="" WHATWEB_QUIET="" # GGGoogleScan settings WAIT="0" DEPTH="5" GSCAN_QUIET="" # Check for WhatWeb in the current working directory WHATWEB="./whatweb" if [[ ! -e "$WHATWEB" ]]; then # Check for WhatWeb directory relative to hunter WHATWEB=`dirname $0`"/../whatweb" if [[ ! -e "$WHATWEB" ]]; then # Check if WhatWeb is installed WHATWEB=`which whatweb 2>/dev/null` if [[ -z "$WHATWEB" ]]; then echo "[!] Fatal Error: WhatWeb must be in the parent directory, in the current working directory or installed. Homepage: http://www.morningstarsecurity.com/research/whatweb" exit 1 fi fi fi # expect gggooglescan to be in the addons/ folder GOOGLESCAN=`dirname "$0"`"/gggooglescan" if [[ ! -e "$GOOGLESCAN" ]]; then echo "[!] Fatal Error: Expected to find gggooglescan in `dirname`. Homepage: http://www.morningstarsecurity.com/research/gggooglescan" exit 1 fi # Show usage and quit function usage { LRED="\033[1;32m" DRED="\033[0;32m" NOCOLOR="\033[0m" echo -e " $LRED @@@ @@@ @@@ @@@ @@@ @@@ @@@@@@@ @@@@@@@@ @@@@@@@ $LRED @@! @@@ @@! @@@ @@!@!@@@ @@! @@! @@! @@@ $DRED @!@!@!@! @!@ !@! @!@@!!@! @!! @!!!:! @!@!!@! $DRED !!: !!! !!: !!! !!: !!! !!: !!: !!: :!! $DRED : : : :.:: : :: : : : :: ::: : : : $NOCOLOR Hunter - Finds web applications with Google then fingerprints them with WhatWeb Version $VERSION by $AUTHOR Usage: ./hunter [options] Example: ./hunter -a wordpress ./hunter -a wordpress -f "wordpress,title,httpserver" ./hunter -g \"site:wordpress.org\" -f wordpress Options: -g QUERY Search Google for QUERY -a APPLICATION Search Google for APPLICATION -f APPLICATION Filter results by comma delimited list of applications -l KEYWORD List supported applications. Filter by KEYWORD. Use \".\" to list all applications. -h This help info -q Quiet. Do not display comment lines and errors to STDOUT Google Options: -d NUM Depth of Google results. NUM pages to return. Default: 5 -w SECONDS Wait for SECONDS between each Google query. Default: 0 WhatWeb Options: -n AGGRESSION Set WhatWeb aggression level. Default: 1 (passive) -u USER-AGENT Set WhatWeb user agent -p Set WhatWeb proxy hostname and port -c Set WhatWeb proxy credentials " } # Command line options while getopts 'd:a:u:f:w:n:g:l:p:c:qh' OPTION do case $OPTION in h) usage; exit ;; d) DEPTH=$OPTARG ;; f) FILTER=$OPTARG ;; n) AGGRESSION=$OPTARG ;; u) USERAGENT=$OPTARG ;; w) WAIT=$OPTARG ;; q) VERBOSE="FALSE" ;; p) PROXY=$OPTARG ;; c) PROXYUSER=$OPTARG ;; l) LIST=$OPTARG ;; g) QUERY=$OPTARG ;; a) APP=$OPTARG ;; esac done shift $(($OPTIND -1 )) # List applications if [[ ! -z "$LIST" ]]; then if [[ "$VERBOSE" == "TRUE" ]]; then echo "[+] Listing applications matching \"$LIST\"" >&2 fi "$WHATWEB" -I | grep "Dorks:" -B 5 | grep -E "^(\w)" | grep -i "$LIST" exit fi # Show usage when no arguments are provided if [[ -z "$APP" && -z "$QUERY" ]]; then usage exit 1 fi # Get dorks from WhatWeb if no Google query was provided if [[ -z "$QUERY" ]]; then # Get dorks from WhatWeb if [[ "$VERBOSE" == "TRUE" ]]; then echo "[-] No query specified. Loading Google queries for $APP" >&2 fi DORKS=`"$WHATWEB" --dorks "$APP"` # Quit if no dorks were returned if [[ -z "$DORKS" ]]; then if [[ "$VERBOSE" == "TRUE" ]]; then echo "[-] No Google queries were found." >&2 fi exit 1 fi else DORKS="$QUERY" fi # Show google dorks if [[ "$VERBOSE" == "TRUE" ]]; then echo "[+] Using the following Google queries:" >&2 echo "$DORKS" fi # Set quiet args if required if [[ "$VERBOSE" == "FALSE" ]]; then WHATWEB_QUIET="-q --no-errors" # gggooglescan's quiet mode is broken in v0.4 if [[ ! -z `grep "getopts" "$GOOGLESCAN" | grep q` ]]; then GSCAN_QUIET="-q" fi fi # Pipe dorks to gggooglescan then pipe the results to WhatWeb echo "$DORKS" | while read LINE ; do "$GOOGLESCAN" $GSCAN_QUIET -d "$DEPTH" -s "$WAIT" "$LINE"; done | egrep -v "^#" | "$WHATWEB" $WHATWEB_QUIET -a "$AGGRESSION" -U "$USERAGENT" -i /dev/stdin --proxy "$PROXY" --proxy-user "$PROXYUSER" --log-brief="$LOG" -p "$FILTER" | grep -e " \[[0-9]" # Show log message if [[ "$VERBOSE" == "TRUE" ]]; then echo "[+] Wrote output to $LOG" >&2 fi WhatWeb-0.4.9/addons/verify-nikto000077500000000000000000000025551320553762300167260ustar00rootroot00000000000000#!/bin/bash # Verify Nikto logs # Copyright 2014, Andrew Horton VERSION=0.1b if [ -z "$1" ]; then echo "Usage: $0 " echo "Version: $VERSION" echo "Verifies Nikto logs using WhatWeb to separate false positives / true positives" exit 1 fi fname="$1" if [ ! -f "$fname" ]; then echo "Cannot read $fname" exit 1 fi tfile=`tempfile --prefix=verify-nikto-` tlog=`tempfile --prefix=whatweb-log-` grep -A 999999 "Start Time:" "$fname" | egrep -o "^+.*(/[^:]+)" | cut -d/ -f 2-|cut -d: -f 1| sed 's/^/\//g' > "$tfile" hostname=`egrep "+ Target (Host|Hostname): " "$fname" | cut -d : -f2 | tr -d ' '` port=`grep "+ Target Port: " "$fname"| cut -d : -f2 | tr -d ' '` if grep -q 'SSL Info' "$fname"; then prefix='https://' else prefix='http://' fi echo "------------------------------------------------------------------------------------------------" echo "Checking Nikto Log: $fname" echo "Running whatweb on $prefix$hostname:$port with `wc -l $tfile` url paths" whatweb --log-brief "$tlog" -p+ -i $tfile --url-prefix "$prefix$hostname:$port" > /dev/null 2>/dev/null echo echo "Different HTML Tag Hashes" echo "------------------------------" egrep -o "Tag-Hash[^ ]+" $tlog|sort -u echo "Plugin analysis" echo -e "Count\tPlugin" echo "------------------------------" egrep -o "[^ ]+\[" "$tlog" |sort|uniq -c | sort -rn echo -e "WhatWeb log: $tlog\nURL path list: $tfile\n" WhatWeb-0.4.9/lib/000077500000000000000000000000001320553762300136415ustar00rootroot00000000000000WhatWeb-0.4.9/lib/colour.rb000066400000000000000000000027731320553762300155020ustar00rootroot00000000000000# colours for consoles =begin Copyright 2009, 2017 Andrew Horton This file is part of WhatWeb. WhatWeb is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or at your option) any later version. WhatWeb is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with WhatWeb. If not, see . =end def colorize(text, color_code) "#{color_code}#{text}\e[0m" end def red(text); colorize(text, "\e[1m\e[31m"); end def dark_red(text); colorize(text, "\e[31m"); end def green(text); colorize(text, "\e[1m\e[32m"); end def dark_green(text); colorize(text, "\e[32m"); end def yellow(text); colorize(text, "\e[1m\e[33m"); end def dark_yellow(text); colorize(text, "\e[33m"); end def blue(text); colorize(text, "\e[1m\e[34m"); end def dark_blue(text); colorize(text, "\e[34m"); end def magenta(text); colorize(text, "\e[1m\e[35m"); end def dark_magenta(text); colorize(text, "\e[35m"); end def cyan(text); colorize(text, "\e[1m\e[36m"); end def dark_cyan(text); colorize(text, "\e[36m"); end def white(text); colorize(text, "\e[1m\e[37m"); end def grey(text); colorize(text, "\e[37m"); end WhatWeb-0.4.9/lib/extend-http_ruby1.9.rb000066400000000000000000000137261320553762300177340ustar00rootroot00000000000000require 'net/protocol' require 'uri' require 'timeout' # # # added @raw # added @real_response # # # Modified HTTP to return the real header block # This *works* with Ruby1.8.7 and Ruby1.9.1 # HTTPS loses SPI # The ExtendedHTTP class is used in place of the HTTP class # for example, # http=ExtendedHTTP.new(@uri.host,@uri.port) # The ExtendedHTTP class uses the ExtendedHTTPResponse class class ExtendedHTTP < Net::HTTP #:nodoc: include Net # class HTTP < Protocol # Creates a new Net::HTTP object for the specified +address+. # This method does not open the TCP connection. # # ExtendedHTTP :: initialize # added @raw=[] def initialize(address, port = nil) @address = address @port = (port || HTTP.default_port) @curr_http_version = HTTPVersion @no_keepalive_server = false @close_on_empty_response = false @socket = nil @started = false @open_timeout = nil @read_timeout = 60 @continue_timeout = nil @debug_output = nil @use_ssl = false @ssl_context = nil @enable_post_connection_check = true @compression = nil @sspi_enabled = false #$DEBUG=true if defined?(SSL_ATTRIBUTES) SSL_ATTRIBUTES.each do |name| instance_variable_set "@#{name}", nil end end # added for whatweb @raw = [] end # ExtendedHTTP :: raw # added def raw def raw @raw end # ExtendedHTTP :: raw # added @raw def connect D "opening connection to #{conn_address()}..." s = timeout(@open_timeout) { TCPSocket.open(conn_address(), conn_port()) } D "opened" if use_ssl? ssl_parameters = Hash.new iv_list = instance_variables SSL_ATTRIBUTES.each do |name| ivname = "@#{name}".intern if iv_list.include?(ivname) and value = instance_variable_get(ivname) ssl_parameters[name] = value end end @ssl_context = OpenSSL::SSL::SSLContext.new @ssl_context.set_params(ssl_parameters) s = OpenSSL::SSL::SSLSocket.new(s, @ssl_context) s.sync_close = true end @socket = BufferedIO.new(s) @socket.read_timeout = @read_timeout @socket.continue_timeout = @continue_timeout @socket.debug_output = @debug_output if use_ssl? begin if proxy? @socket.writeline sprintf('CONNECT %s:%s HTTP/%s', @address, @port, HTTPVersion) @socket.writeline "Host: #{@address}:#{@port}" if proxy_user credential = ["#{proxy_user}:#{proxy_pass}"].pack('m') credential.delete!("\r\n") @socket.writeline "Proxy-Authorization: Basic #{credential}" end @socket.writeline '' # whatweb # HTTPResponse.read_new(@socket).value x,raw = ExtendedHTTPResponse.read_new(@socket) @raw << raw res = x.value end # Server Name Indication (SNI) RFC 3546 s.hostname = @address if s.respond_to? :hostname= timeout(@open_timeout) { s.connect } if @ssl_context.verify_mode != OpenSSL::SSL::VERIFY_NONE s.post_connection_check(@address) end rescue => exception D "Conn close because of connect error #{exception}" @socket.close if @socket and not @socket.closed? raise exception end end on_connect end private :connect def transport_request(req) begin_transport req res = catch(:response) { req.exec @socket, @curr_http_version, edit_path(req.path) begin # added for whatweb #res = HTTPResponse.read_new(@socket) res, y = ExtendedHTTPResponse.read_new(@socket) @raw << y # end while res.kind_of?(HTTPContinue) res.reading_body(@socket, req.response_body_permitted?) { yield res if block_given? } res } end_transport req, res res rescue => exception D "Conn close because of error #{exception}" @socket.close if @socket and not @socket.closed? raise exception end end # added @raw class ExtendedHTTPResponse < Net::HTTPResponse # reopen include Net class << self # read_new and read_status_line are copied from ruby 2 def read_new(sock) #:nodoc: internal use only x,httpv, code, msg = read_status_line(sock) @rawlines= x + "\n" res = response_class(code).new(httpv, code, msg) each_response_header(sock) do |k,v| res.add_field k, v end # added for whatweb real = @rawlines #pp real [res,real] end private def read_status_line(sock) str = sock.readline m = /\AHTTP(?:\/(\d+\.\d+))?\s+(\d\d\d)\s*(.*)\z/in.match(str) or raise HTTPBadResponse, "wrong status line: #{str.dump}" [str] + m.captures end def each_response_header(sock) key = value = nil while true line = sock.readuntil("\n", true).sub(/\s+\z/, '') # added for whatweb @rawlines << line + "\n" unless line.nil? # break if line.empty? if line[0] == ?\s or line[0] == ?\t and value value << ' ' unless value.empty? value << line.strip else yield key, value if key key, value = line.strip.split(/\s*:\s*/, 2) raise Net::HTTPBadResponse, 'wrong header line format' if value.nil? end end yield key, value if key end end public def initialize(httpv, code, msg) #:nodoc: internal use only @http_version = httpv @code = code @message = msg initialize_http_header nil @body = nil @read = false # whatweb @realresponse=[] @rawlines="" end end WhatWeb-0.4.9/lib/extend-http_ruby2.rb000066400000000000000000000166731320553762300175720ustar00rootroot00000000000000require 'net/protocol' require 'uri' require 'timeout' # This works with Ruby 2.x. Based on HTTP library from Ruby 2.2.3p173 # Modified to return the HTTP headers as a string # added @raw # added @raw_lines # # # The ExtendedHTTP class is used in place of the HTTP class # for example, # http=ExtendedHTTP.new(@uri.host,@uri.port) # The ExtendedHTTP class uses the ExtendedHTTPResponse class class ExtendedHTTP < Net::HTTP #:nodoc: include Net # Creates a new Net::HTTP object for the specified server address, # without opening the TCP connection or initializing the HTTP session. # The +address+ should be a DNS hostname or IP address. def initialize(address, port = nil) @address = address @port = (port || HTTP.default_port) @local_host = nil @local_port = nil @curr_http_version = HTTPVersion @keep_alive_timeout = 2 @last_communicated = nil @close_on_empty_response = false @socket = nil @started = false @open_timeout = nil @read_timeout = 60 @continue_timeout = nil @debug_output = nil @proxy_from_env = false @proxy_uri = nil @proxy_address = nil @proxy_port = nil @proxy_user = nil @proxy_pass = nil @use_ssl = false @ssl_context = nil @ssl_session = nil @enable_post_connection_check = true @sspi_enabled = false SSL_IVNAMES.each do |ivname| instance_variable_set ivname, nil end # added for whatweb @raw = [] end # ExtendedHTTP :: raw # added for whatweb def raw @raw end # added @raw for whatweb def connect @raw=[] if proxy? then conn_address = proxy_address conn_port = proxy_port else conn_address = address conn_port = port end D "opening connection to #{conn_address}:#{conn_port}..." s = Timeout.timeout(@open_timeout, Net::OpenTimeout) { TCPSocket.open(conn_address, conn_port, @local_host, @local_port) } s.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1) D "opened" if use_ssl? ssl_parameters = Hash.new iv_list = instance_variables SSL_IVNAMES.each_with_index do |ivname, i| if iv_list.include?(ivname) and value = instance_variable_get(ivname) ssl_parameters[SSL_ATTRIBUTES[i]] = value if value end end @ssl_context = OpenSSL::SSL::SSLContext.new @ssl_context.set_params(ssl_parameters) D "starting SSL for #{conn_address}:#{conn_port}..." s = OpenSSL::SSL::SSLSocket.new(s, @ssl_context) s.sync_close = true D "SSL established" end @socket = BufferedIO.new(s) @socket.read_timeout = @read_timeout @socket.continue_timeout = @continue_timeout @socket.debug_output = @debug_output if use_ssl? begin if proxy? buf = "CONNECT #{@address}:#{@port} HTTP/#{HTTPVersion}\r\n" buf << "Host: #{@address}:#{@port}\r\n" if proxy_user credential = ["#{proxy_user}:#{proxy_pass}"].pack('m') credential.delete!("\r\n") buf << "Proxy-Authorization: Basic #{credential}\r\n" end buf << "\r\n" @socket.write(buf) # HTTPResponse.read_new(@socket).value # added this x,raw=ExtendedHTTPResponse.read_new(@socket) @raw << raw res = x.value # end if @ssl_session and Process.clock_gettime(Process::CLOCK_REALTIME) < @ssl_session.time.to_f + @ssl_session.timeout s.session = @ssl_session if @ssl_session end # Server Name Indication (SNI) RFC 3546 s.hostname = @address if s.respond_to? :hostname= Timeout.timeout(@open_timeout, Net::OpenTimeout) { s.connect } if @ssl_context.verify_mode != OpenSSL::SSL::VERIFY_NONE s.post_connection_check(@address) end @ssl_session = s.session rescue => exception D "Conn close because of connect error #{exception}" @socket.close if @socket and not @socket.closed? raise exception end end on_connect end private :connect def transport_request(req) count = 0 begin begin_transport req res = catch(:response) { req.exec @socket, @curr_http_version, edit_path(req.path) begin # added for whatweb #res = HTTPResponse.read_new(@socket) res, y = ExtendedHTTPResponse.read_new(@socket) @raw << y # res.decode_content = req.decode_content end while res.kind_of?(HTTPContinue) res.uri = req.uri res.reading_body(@socket, req.response_body_permitted?) { yield res if block_given? } res } rescue Net::OpenTimeout raise rescue Net::ReadTimeout, IOError, EOFError, Errno::ECONNRESET, Errno::ECONNABORTED, Errno::EPIPE, # avoid a dependency on OpenSSL defined?(OpenSSL::SSL) ? OpenSSL::SSL::SSLError : IOError, Timeout::Error => exception if count == 0 && IDEMPOTENT_METHODS_.include?(req.method) count += 1 @socket.close if @socket and not @socket.closed? D "Conn close because of error #{exception}, and retry" retry end D "Conn close because of error #{exception}" @socket.close if @socket and not @socket.closed? raise end end_transport req, res res rescue => exception D "Conn close because of error #{exception}" @socket.close if @socket and not @socket.closed? raise exception end end # added @raw class ExtendedHTTPResponse < Net::HTTPResponse # reopen include Net class << self def read_new(sock) #:nodoc: internal use only x,httpv, code, msg = read_status_line(sock) @rawlines= x + "\n" res = response_class(code).new(httpv, code, msg) each_response_header(sock) do |k,v| res.add_field k, v end # added for whatweb real = @rawlines [res,real] end private def read_status_line(sock) str = sock.readline m = /\AHTTP(?:\/(\d+\.\d+))?\s+(\d\d\d)\s*(.*)\z/in.match(str) or raise HTTPBadResponse, "wrong status line: #{str.dump}" [str] + m.captures end def each_response_header(sock) key = value = nil while true line = sock.readuntil("\n", true).sub(/\s+\z/, '') # added for whatweb @rawlines << line + "\n" unless line.nil? # break if line.empty? if line[0] == ?\s or line[0] == ?\t and value value << ' ' unless value.empty? value << line.strip else yield key, value if key key, value = line.strip.split(/\s*:\s*/, 2) raise Net::HTTPBadResponse, 'wrong header line format' if value.nil? end end yield key, value if key end end ################### public # include HTTPHeader def initialize(httpv, code, msg) #:nodoc: internal use only @http_version = httpv @code = code @message = msg initialize_http_header nil @body = nil @read = false @uri = nil @decode_content = false # added for whatweb @rawlines="" end end WhatWeb-0.4.9/lib/http-status.rb000066400000000000000000000062221320553762300164700ustar00rootroot00000000000000=begin Copyright 2009, 2017 Andrew Horton This file is part of WhatWeb. WhatWeb is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or at your option) any later version. WhatWeb is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with WhatWeb. If not, see . =end class HTTP_Status # return HTTP status code as a string def self.code(number) number = number.to_s abort('HTTP_Status must be initialized') unless @status_codes if @status_codes[number] @status_codes[number] else 'Unassigned' end end # Status code list from: # https://raw.githubusercontent.com/Mr-Pi/httpStatusCodes/master/priv/http-status-codes-1.csv def self.initialize @status_codes = {} text = " 100,Continue,[RFC2616] 101,Switching Protocols,[RFC2616] 102,Processing,[RFC2518] 103-199,Unassigned, 200,OK,[RFC2616] 201,Created,[RFC2616] 202,Accepted,[RFC2616] 203,Non-Authoritative Information,[RFC2616] 204,No Content,[RFC2616] 205,Reset Content,[RFC2616] 206,Partial Content,[RFC2616] 207,Multi-Status,[RFC4918] 208,Already Reported,[RFC5842] 209-225,Unassigned, 226,IM Used,[RFC3229] 227-299,Unassigned, 300,Multiple Choices,[RFC2616] 301,Moved Permanently,[RFC2616] 302,Found,[RFC2616] 303,See Other,[RFC2616] 304,Not Modified,[RFC2616] 305,Use Proxy,[RFC2616] 306,Reserved,[RFC2616] 307,Temporary Redirect,[RFC2616] 308,Permanent Redirect,[RFC-reschke-http-status-308-07] 309-399,Unassigned, 400,Bad Request,[RFC2616] 401,Unauthorized,[RFC2616] 402,Payment Required,[RFC2616] 403,Forbidden,[RFC2616] 404,Not Found,[RFC2616] 405,Method Not Allowed,[RFC2616] 406,Not Acceptable,[RFC2616] 407,Proxy Authentication Required,[RFC2616] 408,Request Timeout,[RFC2616] 409,Conflict,[RFC2616] 410,Gone,[RFC2616] 411,Length Required,[RFC2616] 412,Precondition Failed,[RFC2616] 413,Request Entity Too Large,[RFC2616] 414,Request-URI Too Long,[RFC2616] 415,Unsupported Media Type,[RFC2616] 416,Requested Range Not Satisfiable,[RFC2616] 417,Expectation Failed,[RFC2616] 422,Unprocessable Entity,[RFC4918] 423,Locked,[RFC4918] 424,Failed Dependency,[RFC4918] 425,Unassigned, 426,Upgrade Required,[RFC2817] 427,Unassigned, 428,Precondition Required,[RFC6585] 429,Too Many Requests,[RFC6585] 430,Unassigned, 431,Request Header Fields Too Large,[RFC6585] 432-499,Unassigned, 500,Internal Server Error,[RFC2616] 501,Not Implemented,[RFC2616] 502,Bad Gateway,[RFC2616] 503,Service Unavailable,[RFC2616] 504,Gateway Timeout,[RFC2616] 505,HTTP Version Not Supported,[RFC2616] 506,Variant Also Negotiates (Experimental),[RFC2295] 507,Insufficient Storage,[RFC4918] 508,Loop Detected,[RFC5842] 509,Unassigned, 510,Not Extended,[RFC2774] 511,Network Authentication Required,[RFC6585] 512-599,Unassigned, " text.scan(/^([0-9]+),([^,]+)/).each do |k, v| @status_codes[k] = v end end end WhatWeb-0.4.9/lib/output.rb000066400000000000000000001321341320553762300155320ustar00rootroot00000000000000=begin Copyright 2009 to 2017, Andrew Horton This file is part of WhatWeb. WhatWeb is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or at your option) any later version. WhatWeb is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with WhatWeb. If not, see . =end class Output # if no f, output to STDOUT, # if f is a filename then open it, if f is a file use it def initialize(f = STDOUT) f = STDOUT if f == "-" @f = f if f.class == IO or f.class == File @f = File.open(f,"a") if f.class == String @f.sync = true # we want flushed output end def close @f.close unless @f.class == IO end # perform sort, uniq and join on each plugin result def suj(plugin_results) suj = {} [:certainty, :version, :os, :string, :account, :model, :firmware, :module, :filepath].map do |thissymbol| t = plugin_results.map { |x| x[thissymbol] unless x[thissymbol].class == Regexp }.flatten.compact.sort.uniq.join(",") suj[thissymbol] = t end suj[:certainty] = plugin_results.map { |x| x[:certainty] }.flatten.compact.sort.last.to_i # this is different, it's a number suj end # sort and uniq but no join. just for one plugin result def sortuniq(p) su = {} [:name, :certainty, :version, :os, :string, :account, :model, :firmware, :module, :filepath].map do |thissymbol| unless p[thissymbol].class == Regexp t = p[thissymbol] t = t.flatten.compact.sort.uniq if t.is_a?(Array) su[thissymbol] = t unless t.nil? end end # certainty is different, it's a number su[:certainty] = p[:certainty].to_i su end end class OutputObject < Output def out(target, status, results) $semaphore.synchronize do @f.puts "Identifying: #{target.to_s}" @f.puts "HTTP-Status: #{status.to_s}" @f.puts results.pretty_inspect unless results.empty? @f.puts end end end class OutputVerbose < Output def coloured(s, colour) use_colour = ((@f == STDOUT and $use_colour == "auto") or ($use_colour == "always")) if use_colour send colour, s else s end end def out(target, status, results) $semaphore.synchronize do # make a hash of the matches array results_hash={} results.map { |k, v| results_hash[k] = v } display = { title: "", ip: "", country: "", status: "" } display[:country] = results_hash["Country"].map { |r| "#{r[:string]}, #{r[:module]}" }.join(",") if results_hash["Country"] display[:ip] = results_hash["IP"].map { |r| r[:string] }.join(",") if results_hash["Country"] display[:title] = results_hash["Title"].map { |r| r[:string] }.join(",") if results_hash["Title"] display[:status] = "#{status}" + " " + HTTP_Status.code(status) @f.puts "WhatWeb report for #{coloured(target,'blue')}" @f.puts "Status".ljust(9) + " : "+ display[:status] @f.puts "Title".ljust(9) + " : #{coloured(display[:title],'yellow')}" @f.puts "IP".ljust(9) + " : " + display[:ip] @f.puts "Country".ljust(9) + " : #{coloured(display[:country],'red')}" @f.puts ################### Short list ################### Basically Output Brief brief_results = [] results.each do |plugin_name, plugin_results| next if %w[Title IP Country].include? plugin_name unless plugin_results.empty? suj = suj(plugin_results) certainty, version, os, string, accounts, model, firmware, modules, filepath = suj[:certainty].to_i, suj[:version], suj[:os], suj[:string], suj[:account], suj[:model], suj[:firmware], suj[:module], suj[:filepath] # colour the output # be more DRY # if plugins have categories or tags this would be better, eg. all hash plugins are grey if (@f == STDOUT and $use_colour == "auto") or ($use_colour == "always") coloured_string = grey(string) coloured_string = cyan(string) if plugin_name == "HTTPServer" coloured_string = yellow(string) if plugin_name == "Title" coloured_string = grey(string) if plugin_name == "MD5" coloured_string = grey(string) if plugin_name == "Header-Hash" coloured_string = grey(string) if plugin_name == "Footer-Hash" coloured_string = grey(string) if plugin_name == "CSS" coloured_string = grey(string) if plugin_name == "Tag-Hash" coloured_plugin = white(plugin_name) coloured_plugin = grey(plugin_name) if plugin_name == "MD5" coloured_plugin = grey(plugin_name) if plugin_name == "Header-Hash" coloured_plugin = grey(plugin_name) if plugin_name == "Footer-Hash" coloured_plugin = grey(plugin_name) if plugin_name == "CSS" coloured_plugin = grey(plugin_name) if plugin_name == "Tag-Hash" p = ((certainty and certainty < 100) ? "#{grey(certainty_to_words(certainty))} " : "") + coloured_plugin + (!version.empty? ? "[#{green(version)}]" : "") + (!os.empty? ? "[#{red(os)}]" : "") + (!string.empty? ? "[#{coloured_string}]" : "") + (!accounts.empty? ? "[#{cyan(accounts)}]" : "" ) + (!model.empty? ? "[#{dark_green(model)}]" : "" ) + (!firmware.empty? ? "[#{dark_green(firmware)}]" : "" ) + (!filepath.empty? ? "[#{dark_green(filepath)}]" : "" ) + (!modules.empty? ? "[#{magenta(modules)}]" : "" ) brief_results << p else brief_results << ((certainty and certainty < 100) ? "#{certainty_to_words(certainty)} " : "") + plugin_name + (!version.empty? ? "[#{version}]" : "") + (!os.empty? ? "[#{os}]" : "") + (!string.empty? ? "[#{string}]" : "") + (!accounts.empty? ? " [#{accounts}]" : "" ) + (!model.empty? ? "[#{model}]" : "" ) + (!firmware.empty? ? "[#{firmware}]" : "" ) + (!filepath.empty? ? "[#{filepath}]" : "" ) + (!modules.empty? ? "[#{modules}]" : "" ) end end end brief_results_final = brief_results.join(", ") @f.puts "Summary".ljust(9) + " : " + brief_results_final @f.puts @f.puts "Detected Plugins:" results.sort.each do |plugin_name, plugin_results| next if %w[Title IP Country].include?(plugin_name) unless plugin_results.empty? @f.puts "[ #{coloured(plugin_name, 'white')} ]" description = [""] if Plugin.registered_plugins[plugin_name].description d = Plugin.registered_plugins[plugin_name].description description = word_wrap(d, 60) end # @f.puts "\tCategory : " + Plugin.registered_plugins[plugin_name].category.first unless Plugin.registered_plugins[plugin_name].category.nil? @f.puts "\t" + description.first description[1..-1].each { |line| @f.puts "\t" + line } @f.puts top_certainty = suj(plugin_results)[:certainty].to_i unless top_certainty == 100 @f.puts "\t" + "Certainty".ljust(13) + ": " + certainty_to_words(top_certainty) end plugin_results.map { |x| sortuniq(x) }.each do |pr| if pr[:name] name_of_match = pr[:name] else name_of_match = [pr[:regexp_compiled], pr[:text], pr[:regexp].to_s, pr[:ghdb], pr[:md5], pr[:tagpattern]].compact.join("|") end pr.each do |key,value| next unless [:version, :os, :string, :account, :model, :firmware, :module, :filepath, :url].include?(key) next if value.class == Regexp unless key == :os @f.print "\t" + key.to_s.capitalize.ljust(13) + ": " else @f.print "\t" + "OS".ljust(13) + ": " end c = case key when :version then "green" when :string then "cyan" when :certainty then "grey" when :os then "red" when :account then "cyan" when :model then "dark_green" when :firmware then "dark_green" when :module then "magenta" when :filepath then "dark_green" else "grey" end if value.is_a?(String) @f.print coloured(value.to_s,c) elsif value.is_a?(Array) @f.print coloured(value.join(",").to_s, c) else @f.print coloured(value.inspect, c) end unless name_of_match.empty? @f.print " (from #{name_of_match})" end unless pr[:certainty] == 100 @f.print " (Certainty: #{ ertainty_to_words pr[:certainty]} )" end @f.puts end @f.puts "\t" + coloured(pr.inspect.to_s,"dark_blue") if $verbose > 1 end if defined? Plugin.registered_plugins[plugin_name].aggressive @f.puts "\tAggressive function available (check plugin file or details)." end if Plugin.registered_plugins[plugin_name].dorks @f.puts "\tGoogle Dorks".ljust(13)+ ": (#{Plugin.registered_plugins[plugin_name].dorks.size})" end if Plugin.registered_plugins[plugin_name].website @f.puts "\tWebsite".ljust(13)+ ": #{Plugin.registered_plugins[plugin_name].website}" end @f.puts end end @f.puts "HTTP Headers:" target.raw_headers.each_line do |header| @f.puts "\t#{header}" #pp target.raw_headers end end end end class OldOutputVerbose < Output def coloured(s, colour) use_colour = ((@f == STDOUT and $use_colour=="auto") or ($use_colour=="always")) if use_colour send colour, s else s end end def out(target, status, results) $semaphore.synchronize do @f.puts "URL".ljust(7) + ": #{coloured(target,'blue')}" @f.puts "Status".ljust(7) + ": #{status}" results.sort.each do |plugin_name, plugin_results| unless plugin_results.empty? @f.puts " " + coloured(plugin_name, "yellow") + " " + coloured("-"*(80-5-plugin_name.size), "dark_blue") description = [""] unless Plugin.registered_plugins[plugin_name].description.nil? d = Plugin.registered_plugins[plugin_name].description[0..350] d += "..." if d.size == 251 description = word_wrap(d, 60) end # @f.puts "\tCategory : " + Plugin.registered_plugins[plugin_name].category.first unless Plugin.registered_plugins[plugin_name].category.nil? @f.puts "\tDescription: #{description.first}" description[1..-1].each { |line| @f.puts "\t" + " " * 13 + line } unless Plugin.registered_plugins[plugin_name].website.nil? @f.puts "\tWebsite : #{Plugin.registered_plugins[plugin_name].website}" end top_certainty = suj(plugin_results)[:certainty].to_i unless top_certainty == 100 @f.puts "\t" + "Certainty".ljust(11) + ": #{certainty_to_words(top_certainty)}" end plugin_results.map { |x| sortuniq(x) }.each do |pr| if pr[:name] name_of_match = pr[:name] else name_of_match = [pr[:regexp_compiled], pr[:text], pr[:regexp].to_s, pr[:ghdb], pr[:md5], pr[:tagpattern]].compact.join("|") end pr.each do |key, value| next unless [:version, :os, :string, :account, :model, :firmware, :module, :filepath, :url].include?(key) next if value.class == Regexp @f.print "\t" + key.to_s.capitalize.ljust(11) + ": " c = case key when :version then "green" when :string then "cyan" when :certainty then "grey" when :os then "red" when :account then "cyan" when :model then "dark_green" when :firmware then "dark_green" when :module then "magenta" when :filepath then "dark_green" else "grey" end if value.is_a?(String) @f.print coloured(value.to_s, c) elsif value.is_a?(Array) @f.print coloured(value.join(",").to_s, c) else @f.print coloured(value.inspect, c) end unless name_of_match.empty? @f.print " (from #{name_of_match})" end unless pr[:certainty] == 100 @f.print " (Certainty: #{certainty_to_words pr[:certainty]} )" end @f.puts end @f.puts "\t" + coloured(pr.inspect.to_s, "dark_blue") if $verbose > 1 end @f.puts end end end end end class OutputBrief < Output def escape(s) # Encode all special characters # More info: http://www.asciitable.com/ #r=/[^\x20-\x5A\x5E-\x7E]/ # Encode low ascii non printable characters r = /[\x00-\x1F]/ # based on code for CGI.escape s.gsub(r) { |x| "%#{x.unpack('H2' * x.size).join('%').upcase}" } end # don't use colours if not to STDOUT def out(target, status, results) brief_results = [] # sort results so plugins that are less important at a glance are last # last_plugins=%w| CSS MD5 Header-Hash Footer-Hash Tag-Hash| # results=results.sort_by {|x,y| last_plugins.include?(x) ? 1 : 0 } results = results.sort # sort results by plugin name alphabetically results.each do |plugin_name, plugin_results| unless plugin_results.empty? suj = suj(plugin_results) certainty, version, os, string, accounts, model, firmware, modules, filepath = suj[:certainty].to_i, escape(suj[:version]), escape(suj[:os]), escape(suj[:string]), escape(suj[:account]), escape(suj[:model]), escape(suj[:firmware]), escape(suj[:module]), escape(suj[:filepath]) # colour the output # be more DRY # if plugins have categories or tags this would be better, eg. all hash plugins are grey if (@f == STDOUT and $use_colour == "auto") or ($use_colour == "always") coloured_string = grey(string) coloured_string = cyan(string) if plugin_name == "HTTPServer" coloured_string = yellow(string) if plugin_name == "Title" coloured_string = grey(string) if plugin_name == "MD5" coloured_string = grey(string) if plugin_name == "Header-Hash" coloured_string = grey(string) if plugin_name == "Footer-Hash" coloured_string = grey(string) if plugin_name == "CSS" coloured_string = grey(string) if plugin_name == "Tag-Hash" coloured_plugin = white(plugin_name) coloured_plugin = grey(plugin_name) if plugin_name == "MD5" coloured_plugin = grey(plugin_name) if plugin_name == "Header-Hash" coloured_plugin = grey(plugin_name) if plugin_name == "Footer-Hash" coloured_plugin = grey(plugin_name) if plugin_name == "CSS" coloured_plugin = grey(plugin_name) if plugin_name == "Tag-Hash" p = ((certainty and certainty < 100) ? "#{grey(certainty_to_words(certainty))} " : "") + coloured_plugin + (!version.empty? ? "[#{green(version)}]" : "") + (!os.empty? ? "[#{red(os)}]" : "") + (!string.empty? ? "[#{coloured_string}]" : "") + (!accounts.empty? ? "[#{cyan(accounts)}]" : "" ) + (!model.empty? ? "[#{dark_green(model)}]" : "" ) + (!firmware.empty? ? "[#{dark_green(firmware)}]" : "" ) + (!filepath.empty? ? "[#{dark_green(filepath)}]" : "" ) + (!modules.empty? ? "[#{red(modules)}]" : "" ) brief_results << p else brief_results << ((certainty and certainty < 100) ? "#{certainty_to_words(certainty)} " : "") + plugin_name + (!version.empty? ? "[#{version}]" : "") + (!os.empty? ? "[#{os}]" : "") + (!string.empty? ? "[#{string}]" : "") + (!accounts.empty? ? " [#{accounts}]" : "" ) + (!model.empty? ? "[#{model}]" : "" ) + (!firmware.empty? ? "[#{firmware}]" : "" ) + (!filepath.empty? ? "[#{filepath}]" : "" ) + (!modules.empty? ? "[#{modules}]" : "" ) end end end status_code = HTTP_Status.code(status) if (@f == STDOUT and $use_colour=="auto") or ($use_colour=="always") brief_results_final = "#{blue(target)} [#{status} #{status_code}] #{brief_results.join(', ')}" else brief_results_final = "#{target} [#{status} #{status_code}] #{brief_results.join(', ')}" end $semaphore.synchronize do @f.puts brief_results_final end end end # XML Output # # Does anyone use XML output? # We'd love to hear any suggestions you may have! # Does it bother you that some types of output are joined by commas # but other types aren't? class OutputXML < Output def initialize(f=STDOUT) super @substitutions = {'&'=>'&', '"'=>'"', '<'=>'<', '>'=>'>'} # only output ' end @f.puts "" end def close @f.puts "" @f.close end def escape(t) text = t.to_s.dup # use sort_by so that & is before ", etc. @substitutions.sort_by { |a, _| a == "&" ? 0 : 1 }.map{ |from, to| text.gsub!(from, to) } # Encode all special characters # More info: http://www.asciitable.com/ r = /[^\x20-\x5A\x5E-\x7E]/ # based on code for CGI.escape text.gsub!(r) { |x| "%#{x.unpack('H2' * x.size).join('%').upcase}" } text end def out(target, status, results) $semaphore.synchronize do @f.puts "" @f.puts "\t#{escape(target)}" @f.puts "\t#{escape(status)}" @f.puts "\t" @f.puts "\t\t" if $USE_PROXY @f.puts "\t\t\t#{escape($PROXY_HOST)}:#{escape($PROXY_PORT)}" if $PROXY_HOST and $PROXY_PORT @f.puts "\t\t\t#{escape($PROXY_USER)}" if $PROXY_USER @f.puts "\t\t" if $USE_PROXY $CUSTOM_HEADERS.each do |header_name, header_value| @f.puts "\t\t
" @f.puts "\t\t\t#{escape(header_name)}" @f.puts "\t\t\t#{escape(header_value)}" @f.puts "\t\t
" end @f.puts "\t
" results.each do |plugin_name,plugin_results| @f.puts "\t" @f.puts "\t\t#{escape(plugin_name)}" unless plugin_results.empty? # important info in brief mode is version, type and ? # what's the highest probability for the match? certainty = plugin_results.map {|x| x[:certainty] unless x[:certainty].class == Regexp }.flatten.compact.sort.uniq.last version = plugin_results.map {|x| x[:version] unless x[:version].class == Regexp }.flatten.compact.sort.uniq os = plugin_results.map {|x| x[:os] unless x[:os].class == Regexp}.flatten.compact.sort.uniq string = plugin_results.map {|x| x[:string] unless x[:string].class == Regexp}.flatten.compact.sort.uniq model = plugin_results.map {|x| x[:model] unless x[:model].class == Regexp}.flatten.compact.sort.uniq firmware = plugin_results.map {|x| x[:firmware] unless x[:firmware].class == Regexp}.flatten.compact.sort.uniq filepath = plugin_results.map {|x| x[:filepath] unless x[:filepath].class == Regexp}.flatten.compact.sort.uniq account = plugin_results.map {|x| x[:account] unless x[:account].class == Regexp}.flatten.compact.sort.uniq modules = plugin_results.map {|x| x[:module] unless x[:module].class == Regexp}.flatten.compact.sort.uniq # Output results @f.puts "\t\t#{escape(certainty)}" if certainty and certainty < 100 version.map { |x| @f.puts "\t\t#{escape(x)}" } os.map { |x| @f.puts "\t\t#{escape(x)}" } string.map { |x| @f.puts "\t\t#{escape(x)}" } model.map { |x| @f.puts "\t\t#{escape(x)}" } firmware.map { |x| @f.puts "\t\t#{escape(x)}" } filepath.map { |x| @f.puts "\t\t#{escape(x)}" } account.map { |x| @f.puts "\t\t#{escape(x)}" } modules.map { |x| @f.puts "\t\t#{escape(x)}" } end @f.puts "\t" end @f.puts "
" end end end # MagicTree # # Output XML file in MagicTree XML format class OutputMagicTreeXML < Output def initialize(f=STDOUT) super @substitutions={'&'=>'&', '"'=>'"', '<'=>'<', '>'=>'>'} # only output ' end @f.puts '' end def close @f.puts '' @f.close end def escape(t) text = t.to_s.dup # use sort_by so that & is before ", etc. @substitutions.sort_by { |a, _| a == "&" ? 0 : 1 }.map{ |from, to| text.gsub!(from, to) } # Encode all special characters # More info: http://www.asciitable.com/ r = /[^\x20-\x5A\x5E-\x7E]/ # based on code for CGI.escape text.gsub!(r) { |x| "%#{x.unpack('H2' * x.size).join('%').upcase}" } text end def out(target, _status, results) $semaphore.synchronize do # Parse target URL and initialize host node details uri = URI.parse(target.to_s) @host_os = [] @host_port = uri.port @host_scheme = uri.scheme # Set host node details if uri.host =~ /^[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}$/i @host_ip = uri.host @host_name = nil else @host_name = uri.host @host_ip = nil end # Loop through plugin results # get host IP, country and OS results.each do |plugin_name, plugin_results| unless plugin_results.empty? # Host IP @host_ip = plugin_results.map { |x| x[:string] unless x[:string].nil? }.to_s if plugin_name =~ /^IP$/ # Host Country @host_country = plugin_results.map { |x| x[:string] unless x[:string].nil? }.to_s if plugin_name =~ /^Country$/ # Host OS @host_os << plugin_results.map { |x| x[:os] unless x[:os].class == Regexp }.to_s end end # testdata branch @f.write "#{escape(@host_ip)}" # hostname @f.write "#{escape(@host_name)}" unless @host_name.nil? # os @host_os.compact.sort.uniq.map { |x| @f.write "#{escape(x.to_s)}" unless x.empty? } unless @host_os.empty? # country and port nodes @f.write "#{escape(@host_country)}tcp#{escape(@host_port)}open" # https if @host_scheme == 'https' @f.write "ssl"; end # Service node # Loop through remaining results # software, headers, firmware, modules, etc. are all related to a specific URL and therefore are placed under the url node @f.puts "http"; results.each do |plugin_name, plugin_results| if !plugin_results.empty? and plugin_name !~ /^IP$/ and plugin_name !~ /^Country$/ certainty = plugin_results.map { |x| x[:certainty] unless x[:certainty].class == Regexp }.flatten.compact.sort.uniq.last versions = plugin_results.map { |x| x[:version] unless x[:version].class == Regexp }.flatten.compact.sort.uniq strings = plugin_results.map { |x| x[:string] unless x[:string].class == Regexp }.flatten.compact.sort.uniq models = plugin_results.map { |x| x[:model] unless x[:model].class == Regexp }.flatten.compact.sort.uniq firmwares = plugin_results.map { |x| x[:firmware] unless x[:firmware].class == Regexp }.flatten.compact.sort.uniq filepaths = plugin_results.map { |x| x[:filepath] unless x[:filepath].class == Regexp }.flatten.compact.sort.uniq accounts = plugin_results.map { |x| x[:account] unless x[:account].class == Regexp }.flatten.compact.sort.uniq modules = plugin_results.map { |x| x[:module] unless x[:module].class == Regexp }.flatten.compact.sort.uniq # URL node # plugin node @f.write "#{escape(target)}<#{escape(plugin_name)}>" # Print certainty if certainty < 100 if certainty and certainty < 100 @f.write "#{escape(certainty)}" end # Strings if strings.size > 0 strings.map { |x| @f.write escape(x).to_s } unless plugin_name =~ /^IP$/ or plugin_name =~ /^Country$/ end # Versions if versions.size > 0 versions.map { |x| @f.write "#{escape(x)}" } end # Models if models.size > 0 models.map { |x| @f.puts "#{escape(x)}" } end # Firmware if firmwares.size > 0 firmwares.map { |x| @f.write "#{escape(x)}" } end # Modules if modules.size > 0 modules.map { |x| @f.write "#{escape(x)}" } unless plugin_name =~ /^Country$/ end # Accounts # MagicTree generally uses "user" nodes for account if accounts.size > 0 accounts.map { |x| @f.write "#{escape(x)}" } end # Local File Filepaths # Not to be confused with file paths in the web root which are returned in Strings if filepaths.size > 0 filepaths.map { |x| @f.write "#{escape(x)}" } end # debug node # Uncomment to debug # @f.write "Identifying: #{escape(target)}\nHTTP-Status: #{escape(status)}" # @f.write "#{escape(results.pretty_inspect)}" unless results.empty? # @f.write "" # Close plugin name and URL nodes @f.write "" end end @f.write ""; # end https node if @host_scheme == 'https' @f.write "" end # testdata # close port, host and testdata nodes @f.write "" end end end # JSON Output # class OutputJSON < Output def initialize(f = STDOUT) super # opening bracket @f.puts "[" end def close # empty hash because each hash ends with a comma @f.puts "{}" # closing bracket @f.puts "]" @f.close end def flatten_elements!(obj) if obj.class == Hash obj.each_value {|x| flatten_elements!(x) } end if obj.class == Array obj.flatten! end end def utf8_elements!(obj) if obj.class == Hash obj.each_value { |x| utf8_elements!(x) } end if obj.class == Array obj.each { |x| utf8_elements!(x) } end if obj.class == String # obj=obj.upcase! # obj=Iconv.iconv("UTF-8",@charset,obj).join #pp @charset #pp obj.encoding # read this - http://blog.grayproductions.net/articles/ruby_19s_string # replace invalid UTF-8 chars # based on http://stackoverflow.com/a/8873922/388038 if String.method_defined?(:encode) obj.encode!('UTF-16', 'UTF-8', invalid: :replace, replace: '') obj.encode!('UTF-8', 'UTF-16') end obj.force_encoding('UTF-8') # obj=obj.force_encoding("ASCII-8BIT") #puts obj.encoding.name # obj.encode!("UTF-8",{invalid: :replace,undef: :replace}) end end def out(target, status, results) # nice foo= {:target=>target.to_s, :http_status=>status, :request_config=>{}, :plugins=>{} } # request-config req_config = {} if $USE_PROXY req_config[:proxy] = {:proxy_host=>$PROXY_HOST, :proxy_port=>$PROXY_PORT} req_config[:proxy][:proxy_user] = $PROXY_USER if $PROXY_USER end req_config[:headers] = {} unless $CUSTOM_HEADERS.empty? $CUSTOM_HEADERS.each do |header_name, header_value| req_config[:headers][header_name] = header_value.dup end foo[:request_config] = req_config # plugins results.each do |plugin_name,plugin_results| # thisplugin = {name: plugin_name} thisplugin = {} unless plugin_results.empty? # important info in brief mode is version, type and ? # what's the highest probability for the match? certainty = plugin_results.map { |x| x[:certainty] unless x[:certainty].class == Regexp }.flatten.compact.sort.uniq.last version = plugin_results.map { |x| x[:version] unless x[:version].class == Regexp }.flatten.compact.sort.uniq os = plugin_results.map { |x| x[:os] unless x[:os].class == Regexp }.flatten.compact.sort.uniq string = plugin_results.map { |x| x[:string] unless x[:string].class == Regexp }.flatten.compact.sort.uniq accounts = plugin_results.map { |x| x[:account] unless x[:account].class == Regexp }.flatten.compact.sort.uniq model = plugin_results.map { |x| x[:model] unless x[:model].class == Regexp }.flatten.compact.sort.uniq firmware = plugin_results.map { |x| x[:firmware] unless x[:firmware].class == Regexp }.flatten.compact.sort.uniq modules = plugin_results.map { |x| x[:module] unless x[:module].class == Regexp }.flatten.compact.sort.uniq filepath = plugin_results.map { |x| x[:filepath] unless x[:filepath].class == Regexp }.flatten.compact.sort.uniq if !certainty.nil? and certainty != 100 thisplugin[:certainty] = certainty end thisplugin[:version] = version unless version.empty? thisplugin[:os] = os unless os.empty? thisplugin[:string] = string unless string.empty? thisplugin[:account] = accounts unless accounts.empty? thisplugin[:model] = model unless model.empty? thisplugin[:firmware] = firmware unless firmware.empty? thisplugin[:module] = modules unless modules.empty? thisplugin[:filepath] = filepath unless filepath.empty? # foo[:plugins] << thisplugin foo[:plugins][plugin_name.to_sym] = thisplugin end end @charset = results.map { |n, r| r[0][:string] if n == "Charset" }.compact.first unless @charset.nil? or @charset == "Failed" utf8_elements!(foo) # convert foo to utf-8 flatten_elements!(foo) else # could not find encoding force UTF-8 anyway utf8_elements!(foo) end $semaphore.synchronize do @f.puts JSON::generate(foo) + "," end end end # Elasticseach Output, copy of JSON ouput then a HTTP request to send result to elastic class OutputElastic < Output def initialize(s) @host = s[:host] || "127.0.0.1:9200" @index = s[:index] || "whatweb" end def close # nothin' end def flatten_elements!(obj) if obj.class == Hash obj.each_value {|x| flatten_elements!(x) } end if obj.class == Array obj.flatten! end end def utf8_elements!(obj) if obj.class == Hash obj.each_value {|x| utf8_elements!(x) } end if obj.class == Array obj.each {|x| utf8_elements!(x) } end if obj.class == String # obj=obj.upcase! # obj=Iconv.iconv("UTF-8",@charset,obj).join #pp @charset #pp obj.encoding # read this - http://blog.grayproductions.net/articles/ruby_19s_string # replace invalid UTF-8 chars # based on http://stackoverflow.com/a/8873922/388038 if String.method_defined?(:encode) obj.encode!('UTF-16', 'UTF-8', :invalid => :replace, :replace => '') obj.encode!('UTF-8', 'UTF-16') end obj = obj.force_encoding('UTF-8') # obj=obj.force_encoding("ASCII-8BIT") #puts obj.encoding.name # obj.encode!("UTF-8",{:invalid=>:replace,:undef=>:replace}) end end def out(target, status, results) # nice | date be like 2009-11-15T14:12:12 to be autodetected by elastic foo= {:target=>target.to_s, :http_status=>status, :date=>Time.now.strftime('%FT%T'), :plugins=>{} } results.each do |plugin_name,plugin_results| # thisplugin = {:name=>plugin_name} thisplugin = {} unless plugin_results.empty? # important info in brief mode is version, type and ? # what's the highest probability for the match? certainty = plugin_results.map {|x| x[:certainty] unless x[:certainty].class==Regexp }.flatten.compact.sort.uniq.last version = plugin_results.map {|x| x[:version] unless x[:version].class==Regexp }.flatten.compact.sort.uniq os = plugin_results.map {|x| x[:os] unless x[:os].class==Regexp }.flatten.compact.sort.uniq string = plugin_results.map {|x| x[:string] unless x[:string].class==Regexp }.flatten.compact.sort.uniq accounts = plugin_results.map {|x| x[:account] unless x[:account].class==Regexp }.flatten.compact.sort.uniq model = plugin_results.map {|x| x[:model] unless x[:model].class==Regexp }.flatten.compact.sort.uniq firmware = plugin_results.map {|x| x[:firmware] unless x[:firmware].class==Regexp }.flatten.compact.sort.uniq modules = plugin_results.map {|x| x[:module] unless x[:module].class==Regexp }.flatten.compact.sort.uniq filepath = plugin_results.map {|x| x[:filepath] unless x[:filepath].class==Regexp }.flatten.compact.sort.uniq if !certainty.nil? and certainty != 100 thisplugin[:certainty] = certainty end thisplugin[:version] = version unless version.empty? thisplugin[:os] = os unless os.empty? thisplugin[:string] = string unless string.empty? thisplugin[:account] = accounts unless accounts.empty? thisplugin[:model] = model unless model.empty? thisplugin[:firmware] = firmware unless firmware.empty? thisplugin[:module] = modules unless modules.empty? thisplugin[:filepath] = filepath unless filepath.empty? # foo[:plugins] << thisplugin foo[:plugins][plugin_name.to_sym] = thisplugin end end @charset=results.map {|n,r| r[0][:string] if n=="Charset" }.compact.first unless @charset.nil? or @charset == "Failed" utf8_elements!(foo) # convert foo to utf-8 flatten_elements!(foo) else # could not find encoding force UTF-8 anyway utf8_elements!(foo) end url = URI('http://' + @host + '/' + @index + '/whatwebresult') req = Net::HTTP::Post.new(url) req.body = JSON::generate(foo) res = Net::HTTP.start(url.hostname, url.port) {|http| http.request(req) } case res when Net::HTTPSuccess #ok else error(res.code + " " + res.message + "\n" + res.body) end end end # basically the same as OutputJSON class OutputMongo < Output def initialize(s) host = s[:host] || "0.0.0.0" database = s[:database] || raise("Missing MongoDB database name") collection = s[:collection] || "whatweb" # should make databse and collection comma or fullstop delimited, eg. test,scan #@db.authenticate(s[:username], s[:password]) if s[:username] options = { :database => database } if s[:username] options[:username] = s[:username] options[:password] = s[:password] end @db = Mongo::Client.new([ host ], options) @coll = @db[collection] @charset = nil end def close @db.close end def flatten_elements!(obj) if obj.class == Hash obj.each_value {|x| flatten_elements!(x) } end if obj.class == Array obj.flatten! end end def utf8_elements!(obj) if obj.class == Hash obj.each_value {|x| utf8_elements!(x) } end if obj.class == Array obj.each {|x| utf8_elements!(x) } end if obj.class == String # obj=obj.upcase! # obj=Iconv.iconv("UTF-8",@charset,obj).join obj.force_encoding('UTF-8') end end def out(target, status, results) # nice foo= {:target=>target.to_s, :http_status=>status, :request_config=>{}, :plugins=>{} } # request-config req_config = {} if $USE_PROXY req_config[:proxy] = {:proxy_host=>$PROXY_HOST, :proxy_port=>$PROXY_PORT} req_config[:proxy][:proxy_user] = $PROXY_USER if $PROXY_USER end req_config[:headers] = {} unless $CUSTOM_HEADERS.empty? $CUSTOM_HEADERS.each do |header_name, header_value| req_config[:headers][header_name] = header_value.dup end foo[:request_config] = req_config results.each do |plugin_name, plugin_results| # thisplugin = {name: plugin_name} thisplugin = {} unless plugin_results.empty? # important info in brief mode is version, type and ? # what's the highest probability for the match? certainty = plugin_results.map { |x| x[:certainty] unless x[:certainty].class == Regexp }.compact.sort.uniq.last version = plugin_results.map { |x| x[:version] unless x[:version].class == Regexp }.flatten.compact.sort.uniq os = plugin_results.map { |x| x[:os] unless x[:os].class == Regexp }.flatten.compact.sort.uniq string = plugin_results.map { |x| x[:string] unless x[:string].class == Regexp }.flatten.compact.sort.uniq accounts = plugin_results.map { |x| x[:account] unless x[:account].class == Regexp }.flatten.compact.sort.uniq model = plugin_results.map { |x| x[:model] unless x[:model].class == Regexp }.flatten.compact.sort.uniq firmware = plugin_results.map { |x| x[:firmware] unless x[:firmware].class == Regexp }.flatten.compact.sort.uniq modules = plugin_results.map { |x| x[:module] unless x[:module].class == Regexp }.flatten.compact.sort.uniq filepath = plugin_results.map { |x| x[:filepath] unless x[:filepath].class == Regexp }.flatten.compact.sort.uniq if !certainty.nil? and certainty != 100 thisplugin[:certainty] = certainty end thisplugin[:version] = version unless version.empty? thisplugin[:os] = os unless os.empty? thisplugin[:string] = string unless string.empty? thisplugin[:account] = accounts unless accounts.empty? thisplugin[:model] = model unless model.empty? thisplugin[:firmware] = firmware unless firmware.empty? thisplugin[:module] = modules unless modules.empty? thisplugin[:filepath] = filepath unless filepath.empty? # foo[:plugins] << thisplugin foo[:plugins][plugin_name.to_sym] = thisplugin end end @charset = results.map { |n, r| r[0][:string] if n == "Charset" }.compact.first unless @charset.nil? or @charset == "Failed" utf8_elements!(foo) # convert foo to utf-8 flatten_elements!(foo) @coll.insert_one(foo) else error("#{target}: Failed to detect Character set and log to MongoDB") end end end # This is not JSON compliant as a list class OutputJSONVerbose < Output def out(target, status, results) # brutal and simple $semaphore.synchronize do @f.puts JSON::fast_generate([target,status,results]) end end end class OutputErrors < Output # don't need semaphore.synchronize, as it's locked by the error handling routine def out(error) @f.puts error end end class OutputSQL < Output def flatten_elements!(obj) if obj.class == Hash obj.each_value { |x| flatten_elements!(x) } elsif obj.class == Array obj.flatten! end end def escape_for_sql(s) s = s.to_s if s.nil? "''" else "'"+ s.gsub("'","\'")+"'" end end def initialize(f=STDOUT) super end def create_tables # Note that you may encounter the error "1709 - Index column size too large. The maximum column size is 767 bytes." # when using MySQL <= 5.6 with the innodb engine and the utf8mb4 character set # # max_hostname_length = 253 # max_uri_prefix = 10 # covers https:// # max_url_length = 2048 # old IE limit max_target_length = 2048 # feel free to modify this @f.puts "CREATE TABLE plugins (plugin_id int NOT NULL AUTO_INCREMENT, name varchar(255) NOT NULL,PRIMARY KEY (plugin_id), UNIQUE (name));" @f.puts "CREATE TABLE targets (target_id int NOT NULL AUTO_INCREMENT, target varchar(#{max_target_length}) NOT NULL, status varchar(10),PRIMARY KEY (target_id), UNIQUE (target, status) );" @f.puts "CREATE TABLE scans (scan_id int NOT NULL AUTO_INCREMENT, config_id INT NOT NULL, plugin_id INT NOT NULL, target_id INT NOT NULL, version varchar(255), os varchar(255), string varchar(1024), account varchar(1024), model varchar(1024), firmware varchar(1024), module varchar(1024), filepath varchar(1024), certainty varchar(10) ,PRIMARY KEY (scan_id));" @f.puts "CREATE TABLE request_configs (config_id int NOT NULL AUTO_INCREMENT, value TEXT NOT NULL, PRIMARY KEY (config_id) );" # plugins table @f.puts "INSERT INTO plugins (name) VALUES ('Custom-Plugin');" @f.puts "INSERT INTO plugins (name) VALUES ('Grep');" Plugin::registered_plugins.each do |n, _| @f.puts "INSERT INTO plugins (name) VALUES (#{escape_for_sql(n)});" end end def out(target, status, results) # nice foo= {:target=>target, :http_status=>status, :plugins=>{}, :request_config=>{}} # config req_config = {} if $USE_PROXY req_config[:proxy] = {:proxy_host=>$PROXY_HOST, :proxy_port=>$PROXY_PORT} req_config[:proxy][:proxy_user] = $PROXY_USER if $PROXY_USER end req_config[:headers] = {} unless $CUSTOM_HEADERS.empty? $CUSTOM_HEADERS.each do |header_name, header_value| req_config[:headers][header_name] = header_value.dup end foo[:request_config] = req_config results.each do |plugin_name, plugin_results| thisplugin = {} unless plugin_results.empty? # important info in brief mode is version, type and ? # what's the highest probability for the match? certainty = plugin_results.map { |x| x[:certainty] unless x[:certainty].class == Regexp }.flatten.compact.sort.uniq.last version = plugin_results.map { |x| x[:version] unless x[:version].class == Regexp }.flatten.compact.sort.uniq os = plugin_results.map {|x| x[:os] unless x[:os].class == Regexp }.flatten.compact.sort.uniq string = plugin_results.map { |x| x[:string] unless x[:string].class == Regexp }.flatten.compact.sort.uniq accounts = plugin_results.map { |x| x[:account] unless x[:account].class == Regexp }.flatten.compact.sort.uniq model = plugin_results.map { |x| x[:model] unless x[:model].class == Regexp }.flatten.compact.sort.uniq firmware = plugin_results.map { |x| x[:firmware] unless x[:firmware].class == Regexp }.flatten.compact.sort.uniq modules = plugin_results.map { |x| x[:module] unless x[:module].class == Regexp }.flatten.compact.sort.uniq filepath = plugin_results.map { |x| x[:filepath] unless x[:filepath].class == Regexp }.flatten.compact.sort.uniq thisplugin[:certainty] = certainty if !certainty.nil? and certainty != 100 # empty arrays thisplugin[:version] = version.empty? ? [] : version thisplugin[:os] = os.empty? ? [] : os thisplugin[:string] = string.empty? ? [] : string thisplugin[:account] = accounts.empty? ? [] : accounts thisplugin[:model] = model.empty? ? [] : model thisplugin[:firmware] = firmware.empty? ? [] : firmware thisplugin[:module] = modules.empty? ? [] : modules thisplugin[:filepath] = filepath.empty? ? [] : filepath foo[:plugins][plugin_name.to_sym] = thisplugin end end flatten_elements!(foo) i_target = escape_for_sql(foo[:target]) insert = [escape_for_sql(foo[:http_status]), i_target].join(",") query = "INSERT IGNORE INTO targets (status,target) VALUES (#{ insert });"; @f.puts query insert = [escape_for_sql(JSON.dump(foo[:request_config]))].join(",") query = "INSERT INTO request_configs (value) VALUES (#{insert})" @f.puts query foo[:plugins].each do |x| plugin_name = escape_for_sql(x.first.to_s) insert = [escape_for_sql(x[1][:version].join(",").to_s), escape_for_sql(x[1][:os].join(",").to_s), escape_for_sql(x[1][:string].join(",").to_s), escape_for_sql(x[1][:account].join(",").to_s), escape_for_sql(x[1][:model].join(",").to_s), escape_for_sql(x[1][:firmware].join(",").to_s), escape_for_sql(x[1][:module].join(",").to_s), escape_for_sql(x[1][:filepath].join(",").to_s), escape_for_sql(x[1][:certainty].to_s)].join(",") query = "INSERT INTO scans (target_id, config_id, plugin_id, version, os, string, account, model, firmware, module, filepath, certainty) VALUES ( (SELECT target_id from targets WHERE target = #{i_target}),(SELECT MAX(config_id) from request_configs),(SELECT plugin_id from plugins WHERE name = #{plugin_name}), #{insert} );"; @f.puts query end end end WhatWeb-0.4.9/lib/plugins.rb000066400000000000000000000465141320553762300156610ustar00rootroot00000000000000=begin Copyright 2009, 2017 Andrew Horton This file is part of WhatWeb. WhatWeb is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or at your option) any later version. WhatWeb is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with WhatWeb. If not, see . =end module PluginSugar def def_field(*names) class_eval do names.each do |name| define_method(name) do |*args| if args.empty? instance_variable_get("@#{name}") else instance_variable_set("@#{name}", *args) end end end end end end class Plugin extend PluginSugar def_field :author, :version, :description, :website, :matches, :cve, :dorks # deprecated fields def_field :examples #, :category @registered_plugins = {} class << self attr_reader :registered_plugins private :new attr_reader :locked attr_reader :plugin_name @locked=false end def self.define(name, &block) p = new p.set_plugin_name(name) p.instance_eval(&block) p.startup Plugin.registered_plugins[name] = p end def set_plugin_name(s) @plugin_name = s end def version_detection? return false unless @matches !@matches.map { |m| m[:version] }.compact.empty? end # individual plugins can override this def startup; end # individual plugins can override this def shutdown; end def lock @locked=true end def unlock @locked=false end def locked? @locked end def init (target) @target = target @body = target.body @headers = target.headers @status = target.status @base_uri = target.uri @md5sum = target.md5sum @tagpattern = target.tag_pattern @ip = target.ip @raw_response = target.raw_response @raw_headers = target.raw_headers end def make_matches(target, match) r = [] # search location search_context = target.body # by default unless match[:search].nil? case match[:search] when 'all' search_context = target.raw_response when 'headers' search_context = target.raw_headers when /headers\[(.*)\]/ header = $1.downcase if target.headers[header] search_context = target.headers[header] else #error "Invalid search context :search => #{match[:search]}" return r end end end unless match[:ghdb].nil? r << match if match_ghdb(match[:ghdb], target.body, target.headers, target.status, target.uri) end unless match[:text].nil? r << match if match[:regexp_compiled] =~ search_context end unless match[:md5].nil? r << match if target.md5sum == match[:md5] end unless match[:tagpattern].nil? r << match if target.tag_pattern == match[:tagpattern] end if !match[:regexp_compiled].nil? and !search_context.nil? [:regexp, :account, :version, :os, :module, :model, :string, :firmware, :filepath].each do |symbol| if match[symbol] and match[symbol].class==Regexp regexpmatch = search_context.scan(match[:regexp_compiled]) unless regexpmatch.empty? m = match.dup m[symbol] = regexpmatch.map {|eachmatch| if eachmatch.is_a?(Array) and match[:offset] eachmatch[match[:offset]] elsif eachmatch.is_a?(Array) eachmatch.first elsif eachmatch.is_a?(String) eachmatch end }.flatten.compact.sort.uniq r << m end end end end # all previous matches are OR # these are ARE. e.g. required if present return r if r.empty? # if url and status are present, they must both match # url and status cannot be alone. there must be something else that has already matched url_matched = false status_matched = false if match[:status] status_matched = true if match[:status] == target.status end if match[:url] # url is not relative if :url starts with / # url is relative if :url starts with [^/] # url query is only checked if :url has a ? # {:url="edit?action=stop" } will only match if the end of the path and the entire query matches. # :url is for URIs not regexes if match[:url] =~ /^\// is_relative = false else is_relative = true end if match[:url] =~ /\?/ has_query = true else has_query = false end if is_relative and not has_query url_matched = true if target.uri.path =~ /#{match[:url]}$/ end if is_relative and has_query if target.uri.query url_matched = true if "#{target.uri.path}?#{target.uri.query}" =~ /#{match[:url]}$/ end end if not is_relative and has_query if target.uri.query url_matched = true if "#{target.uri.path}?#{target.uri.query}" == match[:url] end end if not is_relative and not has_query url_matched = true if target.uri.path == match[:url] end end # determine whether to return a match if match[:status] and match[:url] if url_matched and status_matched r << match else r = [] end elsif match[:status] and match[:url].nil? if status_matched r << match else r = [] end elsif match[:status].nil? and match[:url] if url_matched r << match else r = [] end elsif match[:status].nil? and match[:url].nil? # nothing to do end r end # execute plugin def x results = [] unless @matches.nil? @matches.each do |match| results += make_matches(@target, match) end end # if the plugin has a passive method, use it results += self.passive if defined? self.passive # if the plugin has an aggressive method and we're in aggressive mode, use it # or if we're guessing all URLs if ($AGGRESSION == 3 and !results.empty?) or ($AGGRESSION == 4) results += self.aggressive if defined? self.aggressive # if any of our matches have a url then fetch it # and check the matches[] # later we can do some caching # we have no caching, so we sort the URLs to fetch and only get 1 unique url per plugin. not great.. unless @matches.nil? lastbase_uri = nil thisstatus, thisurl, thisbody, thisheaders = nil # this shouldn't be necessary but ruby thinks its a local variable to the if end statement @matches.map { |x| x if x[:url] }.compact.sort_by { |x| x[:url] }.map do |match| r = [] # temp results newbase_uri = URI.join(@base_uri.to_s, match[:url]).to_s aggressivetarget = Target.new(newbase_uri) aggressivetarget.open # if $verbose >1 # puts "#{@plugin_name} Aggressive: #{aggressivetarget.uri.to_s} [#{aggressivetarget.status}]" # end results += make_matches(aggressivetarget, match) end end end # clean up results unless results.empty? results.each do |r| r[:certainty] = 100 if r[:certainty].nil? end end results end end # this class contains stuff related to plugins but not necessary to repeat in each plugin we create class PluginSupport # this is used by load_plugins def PluginSupport.load_plugin(f) begin load f rescue => err error("Error: failed to load - #{err}") rescue SyntaxError => err error("Error: Failed to load - #{err}") rescue SystemExit, Interrupt error("ERROR: Failed to load plugins: Interrupted") if $WWDEBUG == true or $verbose > 1 raise end exit 1 end end # precompile regular expressions in plugins for performance def PluginSupport.precompile_regular_expressions Plugin.registered_plugins.each do |thisplugin| matches = thisplugin[1].matches unless matches.nil? matches.each do |thismatch| unless thismatch[:regexp].nil? #pp thismatch thismatch[:regexp_compiled]=Regexp.new(thismatch[:regexp]) end [:version, :os, :string, :account, :model, :firmware, :module, :filepath].each do |label| if !thismatch[label].nil? and thismatch[label].class == Regexp thismatch[:regexp_compiled] = Regexp.new(thismatch[label]) #pp thismatch end end unless thismatch[:text].nil? thismatch[:regexp_compiled] = Regexp.new(Regexp.escape(thismatch[:text])) end end end end end =begin for adding/removing sets of plugins. --plugins +plugins-disabled,-foobar (+ adds to the full set, -removes from the fullset. items can be directories, files or plugin names) --plugins +/tmp/moo.rb --plugins foobar (only select foobar) --plugins ./plugins-disabled,-md5 (select only plugins from the plugins-disabled folder, remove the md5 plugin from the selected list) does not work correctly with mixed plugin names and files =end def PluginSupport.load_plugins(list = nil) # separate list into a and b # a = make list of dir & filenames # b = make list of assumed pluginnames a = [] b = [] plugin_dirs = PLUGIN_DIRS.clone plugin_dirs.map { |p| p = File.expand_path(p) } if list list = list.split(",") plugins_disabled_location = ["plugins-disabled"].map do |x| $LOAD_PATH.map do |y| path = "#{y}/#{x}" path if File.exists?(path) end end.flatten.compact.first list.each { |x| x.gsub!(/^\+$/, "+#{plugins_disabled_location}") } # + is short for +plugins-disabled list.each do |p| choice = PluginChoice.new choice.fill(p) a << choice if choice.type == "file" b << choice if choice.type == "plugin" end #puts "a: list of dir + filenames" #pp a #puts "b: list of plugin names" #pp b #puts "Plugin Dirs" #pp plugin_dirs # sort by neither, add, minus a = a.sort # plugin_dirs gets wiped out if no modifier is used on a file/folder if a.map { |c| c.modifier }.include?(nil) plugin_dirs = [] end minus_files = [] # make list of files not to load a.map { |c| plugin_dirs << c.name if c.modifier.nil? or c.modifier == "+" plugin_dirs -= [c.name] if c.modifier == "-" # for Dirs minus_files << c.name if c.modifier == "-" # for files } #puts "Plugin Dirs" #pp plugin_dirs #puts "before plugin_dirs.each " #pp Plugin.registered_plugins.size # load files from plugin_dirs unless a file is minused plugin_dirs.each do |d| # if a folder, then load all files if File.directory?(d) (Dir.glob("#{d}/*.rb") - minus_files).each { |x| PluginSupport.load_plugin(x) } elsif File.exists?(d) PluginSupport.load_plugin(d) else error("Error: #{d} is not Dir or File") end end #puts "after plugin_dirs.each " #pp Plugin.registered_plugins.size # make list of plugins to run # go through all plugins, remove from list any that match b minus selected_plugin_names = [] if b.map {|c| c.modifier }.include?(nil) selected_plugin_names = [] else selected_plugin_names = Plugin.registered_plugins.map { |n, p| n.downcase } end b.map {|c| selected_plugin_names << c.name if c.modifier.nil? or c.modifier == "+" selected_plugin_names -= [c.name] if c.modifier == "-" } #pp selected_plugin_names # Plugin.registered_plugins is getting wiped out plugins_to_use = Plugin.registered_plugins.map do |n, p| [n, p] if selected_plugin_names.include?(n.downcase) end.compact #puts "after " # report on plugins that couldn't be found unfound_plugins = selected_plugin_names - plugins_to_use.map { |n, p| n.downcase } unless unfound_plugins.empty? puts "Error: The following plugins were not found: " + unfound_plugins.join(",") end else # no selection, so it's default plugin_dirs.each do |d| Dir.glob("#{d}/*.rb").each { |x| PluginSupport.load_plugin(x) } end plugins_to_use = Plugin.registered_plugins end #puts "-" * 80 #pp plugins_to_use plugins_to_use end def PluginSupport.custom_plugin(c, *option) if option == ["grep"] matches = "matches [:text=>\"#{c}\"]" custom = "# coding: ascii-8bit Plugin.define \"Grep\" do author \"Unknown\" description \"User defined\" website \"User defined\" #{matches} end " else # define a custom plugin on the cmdline # ":text=>'powered by abc'" or # "{:text=>'powered by abc'},{:regexp=>/abc [ ]?1/i}" # then it's ok.. if c =~ /:(text|ghdb|md5|regexp|tagpattern)=>[\/'"].*/ matches="matches [\{#{c}\}]" end # this isn't checked for sanity... loading plugins = cmd exec anyway if c =~ /\{.*\}/ matches = "matches [#{c}]" end abort("Invalid custom plugin syntax: #{c}") if matches.nil? custom = "# coding: ascii-8bit Plugin.define \"Custom-Plugin\" do author \"Unknown\" description \"User defined\" website \"User defined\" #{matches} end " end begin # open tmp file f = Tempfile.new('whatweb-custom-plugin') # write f.write(custom) f.close pp custom if $verbose > 2 # load load f.path f.unlink true rescue SyntaxError error("Error: Cannot load custom plugin") false end end ### some UI stuff def PluginSupport.plugin_list terminal_width = 80 puts "WhatWeb Plugin List" puts puts "Plugin Name - Description" puts "-" * terminal_width Plugin.registered_plugins.sort_by { |a, b| a.downcase }.each do |n, p| # output fits more description onto a line line = "#{n} - " line += p.description.delete("\r\n") if p.description if line.size > terminal_width - 1 line = line[0..terminal_width - 4] + "..." end puts line end puts "-" * terminal_width puts puts "Total: #{Plugin.registered_plugins.size} Plugins" puts puts "Hint:" puts "For complete plugin descriptions use : whatweb --info-plugins " puts "Use it without a search term for a complete description of all plugins." puts end # Show Google Dorks def PluginSupport.plugin_dorks(plugin_name) dorks = [] # Loop through plugins Plugin.registered_plugins.each do |n, p| if n.downcase == plugin_name.downcase pp "Google Dorks for #{n}:" if $verbose > 2 dorks << p.dorks unless p.dorks.nil? end end # Show results if present, else show error message if dorks.size > 0 puts dorks else error("Google dork lookup failed: Invalid plugin name or no dorks available") end end # Show plugin information def PluginSupport.plugin_info(keywords = nil) terminal_width = 80 puts "WhatWeb Detailed Plugin List" puts "Searching for " + keywords.join(",") unless keywords.empty? count = {plugins: 0, version_detection: 0, matches: 0, dorks: 0, aggressive: 0, passive: 0 } Plugin.registered_plugins.sort_by { |a, b| a.downcase }.each do |name, plugin| dump = [name, plugin.author, plugin.description, plugin.website, plugin.matches].flatten.compact.to_a.join.downcase # this will fail is an expected variable is not defined or empty if keywords.empty? or keywords.map { |k| dump.include?(k.downcase) }.compact.include?(true) puts "=" * terminal_width puts "Plugin:".ljust(16) + name puts "-" * terminal_width if plugin.description word_wrap(plugin.description, terminal_width - 16).each_with_index do |line, index| if index == 0 print "Description:".ljust(16) else print " " * 16 end puts line end else print "Description:".ljust(16) + "" end puts "Website:".ljust(16) + (plugin.website || "") puts puts "Author:".ljust(16) + (plugin.author || "") puts "Version:".ljust(16) + (plugin.version || "") puts print "Features:".ljust(16) print "[#{defined?(plugin.matches) and plugin.matches ? "Yes" : "No"}]".ljust(7) + "Pattern Matching" if defined?(plugin.matches) and plugin.matches puts " (#{plugin.matches.size})" else puts end puts " " * 16 + "[#{plugin.version_detection? ? "Yes" : "No" }]".ljust(7) + "Version detection from pattern matching" puts " " * 16 + "[#{defined?(plugin.passive) ? "Yes" : "No"}]".ljust(7) + "Function for passive matches" puts " " * 16 + "[#{defined?(plugin.aggressive) ? "Yes" : "No"}]".ljust(7) + "Function for aggressive matches" count[:version_detection] += 1 if plugin.version_detection? count[:passive] += 1 if defined?(plugin.passive) count[:aggressive] += 1 if defined?(plugin.aggressive) print " " * 16 + "[#{plugin.dorks ? "Yes" : "No"}]".ljust(7) + "Google Dorks" if plugin.dorks puts " (#{plugin.dorks.size})" else puts end puts if plugin.dorks puts "Google Dorks:" plugin.dorks.each_with_index do |dork, index| puts "[#{index + 1 }] #{dork}" end puts count[:dorks] += plugin.dorks.size end if defined?(plugin.matches) and plugin.matches #puts "Pattern Matching:" #plugin.matches.each_with_index do |match, index| # puts "[#{index + 1 }] #{match}" #end #puts count[:matches] += plugin.matches.size end count[:plugins] += 1 end end puts "=" * terminal_width puts "Total plugins: #{count[:plugins]}" puts "Total plugins with version detection from pattern matching: #{count[:version_detection]}" puts "Total patterns (regular expressions, text, MD5 hashes, etc): #{count[:matches]}" puts "Total Google dorks: #{count[:dorks]}" puts "Total aggressive functions: #{count[:aggressive]}" puts "Total passive functions: #{count[:passive]}" puts end end # This is used in plugin selection by load_plugins class PluginChoice attr_accessor :modifier, :type, :name def <=>(s) x = -1 if self.modifier.nil? x = 0 if self.modifier == "+" x = 1 if self.modifier == "-" x end def fill(s) self.modifier = nil self.modifier = s[0].chr if ["+", "-"].include?(s[0].chr) if self.modifier self.name = s[1..-1] else self.name = s end # figure out and store the filename or pluginname if File.exists?(File.expand_path(self.name)) self.type = "file" self.name = File.expand_path(self.name) else self.name.downcase! self.type = "plugin" end end end WhatWeb-0.4.9/lib/target.rb000066400000000000000000000206151320553762300154600ustar00rootroot00000000000000=begin Copyright 2009, 2017 Andrew Horton This file is part of WhatWeb. WhatWeb is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or at your option) any later version. WhatWeb is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with WhatWeb. If not, see . =end class Target attr_reader :target attr_reader :uri, :status, :ip, :body, :headers, :raw_headers, :raw_response attr_reader :cookies attr_reader :md5sum attr_reader :tag_pattern attr_reader :is_url, :is_file attr_accessor :http_options @@meta_refresh_regex=/]+content[\s]*=[^>]*[0-9]+;[\s]*url=['"]?([^"'>]+)['"]?[^>]*>/i def inspect # "#{target} " + [@uri,@status,@ip,@body,@headers,@raw_headers,@raw_response,@cookies,@md5sum,@tag_pattern,@is_url,@is_file].join(",") "URI\n#{'*' * 40}\n#{@uri}" + "status\n#{'*' * 40}\n#{@status}" + "ip\n#{'*' * 40}\n#{@ip}" + "header\n#{'*' * 40}\n#{@headers}" + "cookies\n#{'*' * 40}\n#{@cookies}" + "raw_headers\n#{'*' * 40}\n#{@raw_headers}" + "raw_response\n#{'*' * 40}\n#{@raw_response}" + "body\n#{'*' * 40}\n#{@body}" + "md5sum\n#{'*' * 40}\n#{@md5sum}" + "tag_pattern\n#{'*' * 40}\n#{@tag_pattern}" + "is_url\n#{'*' * 40}\n#{@is_url}" + "is_file\n#{'*' * 40}\n#{@is_file}" end def to_s @target end def Target.meta_refresh_regex @@meta_refresh_regex end def is_file? @is_file end def is_url? @is_url end def initialize(target=nil) @target=target @headers={} @http_options={:method => "GET"} # @status=0 if @target =~ /^http[s]?:\/\// @is_url=true else @is_url=false end if File.exists?(@target) @is_file=true if File.directory?(@target) raise "Error: #{@target} is a directory" end if File.readable?(@target) == false raise "Error: You do not have permission to view #{@target}" end else @is_file=false end if self.is_url? @uri=URI.parse(URI.encode(@target)) # is this taking control away from the user? # [400] http://www.alexa.com [200] http://www.alexa.com/ @uri.path = "/" if @uri.path.empty? else # @uri=URI.parse("file://"+@target) @uri=URI.parse("") end end def open if self.is_file? open_file else open_url(@http_options) end ## after open if @body.nil? # Initialize @body variable if the connection is terminated prematurely # This is usually caused by HTTP status codes: 101, 102, 204, 205, 305 @body="" else @md5sum=Digest::MD5.hexdigest(@body) @tag_pattern = make_tag_pattern(@body) if @raw_headers @raw_response = @raw_headers + @body else @raw_response = @body @raw_headers = "" @cookies=[] end end end def open_file begin # target is a file @body=File.open(@target).read # target is a http packet file if @body =~ /^HTTP\/1\.\d [\d]{3} (.+)\r\n\r\n/m # extract http header @headers=Hash.new pageheaders = body.to_s.split(/\r\n\r\n/).first.to_s.split(/\r\n/) @raw_headers = pageheaders.join("\n") + "\r\n\r\n" @status = pageheaders.first.scan(/^HTTP\/1\.\d ([\d]{3}) /).flatten.first.to_i @cookies=[] for k in 1...pageheaders.length section=pageheaders[k].split(/:/).first.to_s.downcase if section =~ /^set-cookie$/i @cookies << pageheaders[k].scan(/:[\s]*(.+)$/).flatten.first else @headers[section]=pageheaders[k].scan(/:[\s]*(.+)$/).flatten.first end end @headers["set-cookie"] = @cookies.join("\n") unless @cookies.nil? or @cookies.empty? # extract html source if @body =~ /^HTTP\/1\.\d [\d]{3} .+?\r\n\r\n(.+)/m @body = @body.scan(/^HTTP\/1\.\d [\d]{3} .+?\r\n\r\n(.+)/m).flatten.first end end rescue => err raise end end def open_url(options) begin if $USE_PROXY == true http=ExtendedHTTP::Proxy($PROXY_HOST, $PROXY_PORT, $PROXY_USER, $PROXY_PASS).new(@uri.host, @uri.port) else http=ExtendedHTTP.new(@uri.host, @uri.port) end # set timeouts http.open_timeout = $HTTP_OPEN_TIMEOUT http.read_timeout = $HTTP_READ_TIMEOUT # if it's https:// # i wont worry about certificates, verfication, etc if @uri.class == URI::HTTPS http.use_ssl = true OpenSSL::SSL::SSLContext::DEFAULT_PARAMS[:ciphers] = "TLSv1:TLSv1.1:TLSv1.2:SSLv3:SSLv2" http.verify_mode = OpenSSL::SSL::VERIFY_NONE end getthis = @uri.path + (@uri.query.nil? ? "" : "?" + @uri.query) req=nil if options[:method] == "GET" req=ExtendedHTTP::Get.new(getthis, $CUSTOM_HEADERS) end if options[:method] == "HEAD" req=ExtendedHTTP::Head.new(getthis, $CUSTOM_HEADERS) end if options[:method] == "POST" req=ExtendedHTTP::Post.new(getthis, $CUSTOM_HEADERS) req.set_form_data(options[:data]) end if $BASIC_AUTH_USER req.basic_auth $BASIC_AUTH_USER, $BASIC_AUTH_PASS end res=http.request(req) @raw_headers=http.raw.join("\n") @headers={}; res.each_header { |x, y| @headers[x]=y } @headers["set-cookie"] = res.get_fields('set-cookie').join("\n") unless @headers["set-cookie"].nil? @body=res.body @status=res.code.to_i puts @uri.to_s + " [#{status}]" if $verbose > 1 rescue SocketError => err error(@target + " ERROR: Socket error #{err}") return rescue Timeout::Error => err error(@target + " ERROR: Timed out #{err}") return rescue Errno::ETIMEDOUT => err # for ruby 1.8.7 patch level 249 error(@target + " ERROR: Timed out (ETIMEDOUT) #{err}") return rescue EOFError => err error(@target + " ERROR: EOF error #{err}") return rescue StandardError => err err = "Not HTTP or cannot resolve hostname" if err.to_s == "undefined method `closed?' for nil:NilClass" error(@target + " ERROR: #{err}") return rescue => err error(@target + " ERROR: #{err}") return end begin @ip = Resolv.getaddress(@uri.host) rescue StandardError => err err = "Cannot resolve hostname" if err.to_s == "undefined method `closed?' for nil:NilClass" error(@target + " ERROR: #{err}") return end end def get_redirection_target newtarget_m=nil newtarget_h=nil newtarget=nil if @@meta_refresh_regex =~ @body metarefresh=@body.scan(@@meta_refresh_regex).flatten.first metarefresh=decode_html_entities(metarefresh) newtarget_m=URI.join(@target, metarefresh).to_s # this works for relative and absolute end # HTTP 3XX redirect if (300..399) === @status and @headers and @headers['location'] # downcase location scheme location = @headers["location"].gsub(/^HTTPS:\/\//, 'https://').gsub(/^HTTP:\/\//, 'http://') newtarget_h=URI.join(@target,location).to_s end # if both meta refresh location and HTTP location are set, then the HTTP location overrides if newtarget_m or newtarget_h case $FOLLOW_REDIRECT when "never" no_redirects=true # this never gets back to main loop but no prob when "http-only" newtarget = newtarget_h when "meta-only" newtarget = newtarget_m when "same-site" newtarget = (newtarget_h or newtarget_m) if URI.parse((newtarget_h or newtarget_m)).host == @uri.host # defaults to _h if both are present when "same-domain" newtarget = (newtarget_h or newtarget_m) if TLD.same_domain?( @uri.host, URI.parse((newtarget_h or newtarget_m)).host) when "always" newtarget = (newtarget_h or newtarget_m) else error("Error: Invalid REDIRECT mode") end end newtarget=nil if newtarget == @uri.to_s # circular redirection not allowed newtarget end end WhatWeb-0.4.9/lib/tld.rb000066400000000000000000001267301320553762300147620ustar00rootroot00000000000000#!/bin/env ruby # encoding: utf-8 # coding: utf-8 # # Copyright Andrew Horton, 2010 # Permission is granted for use of this source code to be used within WhatWeb # note that is out-dated class TLD attr_reader :tlds # test domains such as XN--0ZWM56D are not included @tlds=["biz","com","info","name","net","org","pro", # generic "aero","asia","cat","coop","edu","gov","int","jobs","mil","mobi","museum","tel","travel", "xxx", #sponsored "arpa", #infrastructure "ac","ad","ae","af","ag","ai","al","am","an","ao","aq","ar","as","at","au","aw","ax","az","ba","bb","bd","be","bf", "bg","bh","bi","bj","bm","bn","bo","br","bs","bt","bv","bw","by","bz","ca","cc","cd","cf","cg","ch","ci","ck","cl", "cm","cn","co","cr","cu","cv","cx","cy","cz","de","dj","dk","dm","do","dz","ec","ee","eg","eh","er","es","et","eu", "fi","fj","fk","fm","fo","fr","ga","gb","gd","ge","gf","gg","gh","gi","gl","gm","gn","gp","gq","gr","gs","gt","gu", "gw","gy","hk","hm","hn","hr","ht","hu","id","ie","il","im","in","io","iq","ir","is","it","je","jm","jo","jp","ke", "kg","kh","ki","km","kn","kp","kr","kw","ky","kz","la","lb","lc","li","lk","lr","ls","lt","lu","lv","ly","ma","mc", "md","me","mg","mh","mk","ml","mm","mn","mo","mp","mq","mr","ms","mt","mu","mv","mw","mx","my","mz","na","nc","ne", "nf","ng","ni","nl","no","np","nr","nu","nz","om","pa","pe","pf","pg","ph","pk","pl","pm","pn","pr","ps","pt","pw", "py","qa","re","ro","rs","ru","rw","sa","sb","sc","sd","se","sg","sh","si","sj","sk","sl","sm","sn","so","sr","st", "su","sv","sy","sz","tc","td","tf","tg","th","tj","tk","tl","tm","tn","to","tp","tr","tt","tv","tw","tz","ua","ug", "uk","us","uy","uz","va","vc","ve","vg","vi","vn","vu","wf","ws","ye","yt","za","zm","zw" #countries ] # if you can register at the 2nd level, then sld's arent listed. # the point of this is to identify & remove domains that aren't possible. # also used to work out pluralisation - may miss some slds not listed # would be more complete with antoehr variable to say if 2nd level reg is possible and have slds for all countries @tld={ "biz"=>{"type"=>"tld","tld"=>"biz","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "com"=>{"type"=>"tld","tld"=>"com","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "info"=>{"type"=>"tld","tld"=>"info","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "name"=>{"type"=>"tld","tld"=>"name","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "net"=>{"type"=>"tld","tld"=>"net","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "org"=>{"type"=>"tld","tld"=>"org","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "pro"=>{"type"=>"tld","tld"=>"pro","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "aero"=>{"type"=>"tld","tld"=>"aero","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "asia"=>{"type"=>"tld","tld"=>"asia","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "cat"=>{"type"=>"tld","tld"=>"cat","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "coop"=>{"type"=>"tld","tld"=>"coop","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "edu"=>{"type"=>"tld","tld"=>"edu","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"US","slds"=>[]}, "gov"=>{"type"=>"tld","tld"=>"gov","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"US","slds"=>[]}, "int"=>{"type"=>"tld","tld"=>"int","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "jobs"=>{"type"=>"tld","tld"=>"jobs","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "mil"=>{"type"=>"tld","tld"=>"mil","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Intl","slds"=>[]}, "mobi"=>{"type"=>"tld","tld"=>"mobi","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "museum"=>{"type"=>"tld","tld"=>"museum","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "tel"=>{"type"=>"tld","tld"=>"tel","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "travel"=>{"type"=>"tld","tld"=>"travel","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "arpa"=>{"type"=>"tld","tld"=>"arpa","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Intl","slds"=>[]}, "ac"=>{"type"=>"country","tld"=>"ac","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Ascension Island","slds"=>[]}, "ad"=>{"type"=>"country","tld"=>"ad","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Andorra","slds"=>[]}, "ae"=>{"type"=>"country","tld"=>"ae","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"United Arab Emirates","slds"=>[]}, "af"=>{"type"=>"country","tld"=>"af","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Afghanistan","slds"=>[]}, "ag"=>{"type"=>"country","tld"=>"ag","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Antigua and Barbuda","slds"=>[]}, "ai"=>{"type"=>"country","tld"=>"ai","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Anguilla","slds"=>[]}, "al"=>{"type"=>"country","tld"=>"al","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Albania","slds"=>["gov.al","edu.al","org.al","com.al","net.al"]}, "am"=>{"type"=>"country","tld"=>"am","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Armenia","slds"=>[]}, "an"=>{"type"=>"country","tld"=>"an","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Netherlands Antilles","slds"=>[]}, "ao"=>{"type"=>"country","tld"=>"ao","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Angola","slds"=>["ed.ao","gv.ao","og.ao","co.ao","pb.ao","it.ao"]}, "aq"=>{"type"=>"country","tld"=>"aq","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Antarctica","slds"=>[]}, "ar"=>{"type"=>"country","tld"=>"ar","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Argentina","slds"=>["com.ar","edu.ar","gob.ar","gov.ar","int.ar","mil.ar","net.ar","org.ar","tur.ar"]}, "as"=>{"type"=>"country","tld"=>"as","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"American Samoa","slds"=>[]}, "at"=>{"type"=>"country","tld"=>"at","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Austria","slds"=>[]}, "au"=>{"type"=>"country","tld"=>"au","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Australia","slds"=>["com.au","net.au","org.au","edu.au","gov.au","csiro.au","asn.au","id.au","act.au","nsw.au","nt.au","qld.au","sa.au","tas.au","vic.au","wa.au"]}, "aw"=>{"type"=>"country","tld"=>"aw","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Aruba","slds"=>[]}, "ax"=>{"type"=>"country","tld"=>"ax","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Åland Islands","slds"=>[]}, "az"=>{"type"=>"country","tld"=>"az","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Azerbaijan","slds"=>[]}, "ba"=>{"type"=>"country","tld"=>"ba","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Bosnia and Herzegovina","slds"=>[]}, "bb"=>{"type"=>"country","tld"=>"bb","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Barbados","slds"=>[]}, "bd"=>{"type"=>"country","tld"=>"bd","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Bangladesh","slds"=>["com.bd","edu.bd","ac.bd","net.bd","gov.bd","org.bd","mil.bd"]}, "be"=>{"type"=>"country","tld"=>"be","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Belgium","slds"=>[]}, "bf"=>{"type"=>"country","tld"=>"bf","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Burkina Faso","slds"=>[]}, "bg"=>{"type"=>"country","tld"=>"bg","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Bulgaria","slds"=>[]}, "bh"=>{"type"=>"country","tld"=>"bh","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Bahrain","slds"=>[]}, "bi"=>{"type"=>"country","tld"=>"bi","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Burundi","slds"=>[]}, "bj"=>{"type"=>"country","tld"=>"bj","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Benin","slds"=>[]}, "bm"=>{"type"=>"country","tld"=>"bm","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Bermuda","slds"=>[]}, "bn"=>{"type"=>"country","tld"=>"bn","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Brunei","slds"=>["com.bn","edu.bn","org.bn","net.bn"]}, "bo"=>{"type"=>"country","tld"=>"bo","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Bolivia","slds"=>[]}, "br"=>{"type"=>"country","tld"=>"br","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Brazil","slds"=>["adm.br","adv.br","agr.br","am.br","arq.br","art.br","ato.br","bio.br","blog.br","bmd.br","cim.br","cng.br","cnt.br","com.br","coop.br","ecn.br","edu.br","eng.br","esp.br","etc.br","eti.br","far.br","flog.br","fm.br","fnd.br","fot.br","fst.br","g12.br","ggf.br","gov.br","imb.br","ind.br","inf.br","jor.br","jus.br","lel.br","mat.br","med.br","mil.br","mus.br","net.br","nom.br","not.br","ntr.br","odo.br","org.br","ppg.br","pro.br","psc.br","psi.br","qsl.br","rec.br","slg.br","srv.br","tmp.br","trd.br","tur.br","tv.br","vet.br","vlog.br","wiki.br","zlg.br"]}, "bs"=>{"type"=>"country","tld"=>"bs","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Bahamas","slds"=>[]}, "bt"=>{"type"=>"country","tld"=>"bt","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Bhutan","slds"=>[]}, # {"type"=>"country","tld"=>"bv","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Bouvet Island","slds"=>[]}, "bw"=>{"type"=>"country","tld"=>"bw","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Botswana","slds"=>[]}, "by"=>{"type"=>"country","tld"=>"by","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Belarus","slds"=>[]}, "bz"=>{"type"=>"country","tld"=>"bz","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Belize","slds"=>[]}, "ca"=>{"type"=>"country","tld"=>"ca","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Canada","slds"=>[]}, "cc"=>{"type"=>"country","tld"=>"cc","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Cocos Islands","slds"=>[]}, "cd"=>{"type"=>"country","tld"=>"cd","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Democratic Republic of the Congo","slds"=>[]}, "cf"=>{"type"=>"country","tld"=>"cf","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Central African Republic","slds"=>[]}, "cg"=>{"type"=>"country","tld"=>"cg","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Republic of the Congo","slds"=>[]}, "ch"=>{"type"=>"country","tld"=>"ch","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Switzerland","slds"=>[]}, "ci"=>{"type"=>"country","tld"=>"ci","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Côte d'Ivoire","slds"=>[]}, "ck"=>{"type"=>"country","tld"=>"ck","2nd_level_registration"=>false,"foreign_registration"=>true,"country"=>"Cook Islands","slds"=>["co.ck","org.ck","edu.ck","gov.ck","net.ck","gen.ck","biz.ck","info.ck"]}, "cl"=>{"type"=>"country","tld"=>"cl","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Chile","slds"=>[]}, "cm"=>{"type"=>"country","tld"=>"cm","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Cameroon","slds"=>[]}, "cn"=>{"type"=>"country","tld"=>"cn","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"People's Republic of China","slds"=>[]}, "co"=>{"type"=>"country","tld"=>"co","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Colombia","slds"=>["com.co","org.co","edu.co","gov.co","net.co","mil.co","nom.co"]}, "cr"=>{"type"=>"country","tld"=>"cr","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Costa Rica","slds"=>[]}, "cu"=>{"type"=>"country","tld"=>"cu","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Cuba","slds"=>[]}, "cv"=>{"type"=>"country","tld"=>"cv","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Cape Verde","slds"=>[]}, "cx"=>{"type"=>"country","tld"=>"cx","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Christmas Island","slds"=>[]}, "cy"=>{"type"=>"country","tld"=>"cy","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Cyprus","slds"=>["ac.cy","net.cy","gov.cy","org.cy","pro.cy","name.cy","ekloges.cy","tm.cy","ltd.cy","biz.cy","press.cy","parliament.cy","com.cy"]}, "cz"=>{"type"=>"country","tld"=>"cz","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Czech Republic","slds"=>[]}, "de"=>{"type"=>"country","tld"=>"de","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Germany","slds"=>[]}, "dj"=>{"type"=>"country","tld"=>"dj","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Djibouti","slds"=>[]}, "dk"=>{"type"=>"country","tld"=>"dk","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Denmark","slds"=>[]}, "dm"=>{"type"=>"country","tld"=>"dm","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Dominica","slds"=>[]}, "do"=>{"type"=>"country","tld"=>"do","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Dominican Republic","slds"=>["edu.do","gob.do","com.do","sld.do","org.do","net.do","web.do","mil.do","art.do"]}, "dz"=>{"type"=>"country","tld"=>"dz","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Algeria","slds"=>[]}, "ec"=>{"type"=>"country","tld"=>"ec","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Ecuador","slds"=>[]}, "ee"=>{"type"=>"country","tld"=>"ee","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Estonia","slds"=>[]}, "eg"=>{"type"=>"country","tld"=>"eg","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Egypt","slds"=>["com.eg","edu.eg","eun.eg","gov.eg","mil.eg","name.eg","net.eg","org.eg","sci.eg"]}, # {"type"=>"country","tld"=>"eh","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Western Sahara","slds"=>[]}, "er"=>{"type"=>"country","tld"=>"er","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Eritrea","slds"=>["com.er","edu.er","gov.er","mil.er","net.er","org.er","ind.er","rochest.er","w.er"]}, "es"=>{"type"=>"country","tld"=>"es","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Spain","slds"=>[]}, "et"=>{"type"=>"country","tld"=>"et","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Ethiopia","slds"=>["com.et","gov.et","org.et","edu.et","net.et","biz.et","name.et","info.et"]}, "eu"=>{"type"=>"country","tld"=>"eu","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"European Union","slds"=>[]}, "fi"=>{"type"=>"country","tld"=>"fi","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Finland","slds"=>[]}, "fj"=>{"type"=>"country","tld"=>"fj","2nd_level_registration"=>false,"foreign_registration"=>true,"country"=>"Fiji","slds"=>["ac.fj","biz.fj","com.fj","info.fj","mil.fj","name.fj","net.fj","org.fj","pro.fj"]}, "fk"=>{"type"=>"country","tld"=>"fk","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Falkland Islands","slds"=>["co.fk","org.fk","gov.fk","ac.fk","nom.fk","net.fk"]}, "fm"=>{"type"=>"country","tld"=>"fm","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Federated States of Micronesia","slds"=>[]}, "fo"=>{"type"=>"country","tld"=>"fo","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Faroe Islands","slds"=>[]}, "fr"=>{"type"=>"country","tld"=>"fr","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"France","slds"=>[]}, "ga"=>{"type"=>"country","tld"=>"ga","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Gabon","slds"=>[]}, # {"type"=>"country","tld"=>"gb","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"United Kingdom","slds"=>[]} "gd"=>{"type"=>"country","tld"=>"gd","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Grenada","slds"=>[]}, "ge"=>{"type"=>"country","tld"=>"ge","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Georgia","slds"=>[]}, "gf"=>{"type"=>"country","tld"=>"gf","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"French Guiana","slds"=>[]}, "gg"=>{"type"=>"country","tld"=>"gg","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Guernsey","slds"=>[]}, "gh"=>{"type"=>"country","tld"=>"gh","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Ghana","slds"=>["com.gh","edu.gh","gov.gh","org.gh","mil.gh"]}, "gi"=>{"type"=>"country","tld"=>"gi","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Gibraltar","slds"=>[]}, "gl"=>{"type"=>"country","tld"=>"gl","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Greenland","slds"=>[]}, "gm"=>{"type"=>"country","tld"=>"gm","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Gambia","slds"=>[]}, "gn"=>{"type"=>"country","tld"=>"gn","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Guinea","slds"=>["com.gn","ac.gn","gov.gn","org.gn","net.gn"]}, "gp"=>{"type"=>"country","tld"=>"gp","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Guadeloupe","slds"=>[]}, "gq"=>{"type"=>"country","tld"=>"gq","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Equatorial Guinea","slds"=>[]}, "gr"=>{"type"=>"country","tld"=>"gr","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Greece","slds"=>[]}, "gs"=>{"type"=>"country","tld"=>"gs","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"South Georgia and the South Sandwich Islands","slds"=>[]}, "gt"=>{"type"=>"country","tld"=>"gt","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Guatemala","slds"=>[]}, "gu"=>{"type"=>"country","tld"=>"gu","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Guam","slds"=>[]}, "gw"=>{"type"=>"country","tld"=>"gw","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Guinea-Bissau","slds"=>[]}, "gy"=>{"type"=>"country","tld"=>"gy","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Guyana","slds"=>[]}, "hk"=>{"type"=>"country","tld"=>"hk","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Hong Kong","slds"=>[]}, "hm"=>{"type"=>"country","tld"=>"hm","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Heard Island and McDonald Islands","slds"=>[]}, "hn"=>{"type"=>"country","tld"=>"hn","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Honduras","slds"=>[]}, "hr"=>{"type"=>"country","tld"=>"hr","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Croatia","slds"=>[]}, "ht"=>{"type"=>"country","tld"=>"ht","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Haiti","slds"=>[]}, "hu"=>{"type"=>"country","tld"=>"hu","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Hungary","slds"=>[]}, "id"=>{"type"=>"country","tld"=>"id","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Indonesia","slds"=>["ac.id","co.id","net.id","or.id","web.id","sch.id","mil.id","go.id"]}, "ie"=>{"type"=>"country","tld"=>"ie","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Ireland","slds"=>[]}, "il"=>{"type"=>"country","tld"=>"il","2nd_level_registration"=>false,"foreign_registration"=>true,"country"=>"Israel","slds"=>["ac.il","co.il","org.il","net.il","k12.il","gov.il","muni.il","idf.il"]}, "im"=>{"type"=>"country","tld"=>"im","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Isle of Man","slds"=>[]}, "in"=>{"type"=>"country","tld"=>"in","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"India","slds"=>[]}, "io"=>{"type"=>"country","tld"=>"io","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"British Indian Ocean Territory","slds"=>[]}, "iq"=>{"type"=>"country","tld"=>"iq","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Iraq","slds"=>[]}, "ir"=>{"type"=>"country","tld"=>"ir","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Iran","slds"=>[]}, "is"=>{"type"=>"country","tld"=>"is","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Iceland","slds"=>[]}, "it"=>{"type"=>"country","tld"=>"it","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Italy","slds"=>[]}, "je"=>{"type"=>"country","tld"=>"je","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Jersey","slds"=>[]}, "jm"=>{"type"=>"country","tld"=>"jm","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Jamaica","slds"=>["com.jm","net.jm","org.jm","edu.jm","gov.jm","mil.jm"]}, "jo"=>{"type"=>"country","tld"=>"jo","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Jordan","slds"=>[]}, "jp"=>{"type"=>"country","tld"=>"jp","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Japan","slds"=>[]}, "ke"=>{"type"=>"country","tld"=>"ke","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Kenya","slds"=>["co.ke","or.ke","ne.ke","go.ke","ac.ke","sc.ke"]}, "kg"=>{"type"=>"country","tld"=>"kg","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Kyrgyzstan","slds"=>[]}, "kh"=>{"type"=>"country","tld"=>"kh","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Cambodia","slds"=>["per.kh","com.kh","edu.kh","gov.kh","mil.kh","net.kh","org.kh"]}, "ki"=>{"type"=>"country","tld"=>"ki","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Kiribati","slds"=>[]}, "km"=>{"type"=>"country","tld"=>"km","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Comoros","slds"=>[]}, "kn"=>{"type"=>"country","tld"=>"kn","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Saint Kitts and Nevis","slds"=>[]}, "kp"=>{"type"=>"country","tld"=>"kp","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"North Korea","slds"=>[]}, "kr"=>{"type"=>"country","tld"=>"kr","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"South Korea","slds"=>[]}, "kw"=>{"type"=>"country","tld"=>"kw","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Kuwait","slds"=>["edu.kw","com.kw","net.kw","org.kw","gov.kw"]}, "ky"=>{"type"=>"country","tld"=>"ky","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Cayman Islands","slds"=>[]}, "kz"=>{"type"=>"country","tld"=>"kz","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Kazakhstan","slds"=>[]}, "la"=>{"type"=>"country","tld"=>"la","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Laos","slds"=>[]}, "lb"=>{"type"=>"country","tld"=>"lb","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Lebanon","slds"=>["com.lb","edu.lb","gov.lb","net.lb","org.lb"]}, "lc"=>{"type"=>"country","tld"=>"lc","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Saint Lucia","slds"=>[]}, "li"=>{"type"=>"country","tld"=>"li","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Liechtenstein","slds"=>[]}, "lk"=>{"type"=>"country","tld"=>"lk","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Sri Lanka","slds"=>[]}, "lr"=>{"type"=>"country","tld"=>"lr","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Liberia","slds"=>["com.lr","edu.lr","gov.lr","org.lr","net.lr"]}, "ls"=>{"type"=>"country","tld"=>"ls","2nd_level_registration"=>false,"foreign_registration"=>true,"country"=>"Lesotho","slds"=>["co.ls","org.ls"]}, "lt"=>{"type"=>"country","tld"=>"lt","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Lithuania","slds"=>[]}, "lu"=>{"type"=>"country","tld"=>"lu","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Luxembourg","slds"=>[]}, "lv"=>{"type"=>"country","tld"=>"lv","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Latvia","slds"=>[]}, "ly"=>{"type"=>"country","tld"=>"ly","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Libya","slds"=>[]}, "ma"=>{"type"=>"country","tld"=>"ma","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Morocco","slds"=>[]}, "mc"=>{"type"=>"country","tld"=>"mc","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Monaco","slds"=>[]}, "md"=>{"type"=>"country","tld"=>"md","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Moldova","slds"=>[]}, "me"=>{"type"=>"country","tld"=>"me","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Montenegro","slds"=>[]}, "mg"=>{"type"=>"country","tld"=>"mg","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Madagascar","slds"=>[]}, "mh"=>{"type"=>"country","tld"=>"mh","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Marshall Islands","slds"=>[]}, "mk"=>{"type"=>"country","tld"=>"mk","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Republic of Macedonia","slds"=>[]}, "ml"=>{"type"=>"country","tld"=>"ml","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Mali","slds"=>["com.ml","net.ml","org.ml","edu.ml","gov.ml","presse.ml"]}, "mm"=>{"type"=>"country","tld"=>"mm","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Myanmar","slds"=>["net.mm","com.mm","org.mm","edu.mm","gov.mm"]}, "mn"=>{"type"=>"country","tld"=>"mn","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Mongolia","slds"=>[]}, "mo"=>{"type"=>"country","tld"=>"mo","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Macau","slds"=>[]}, "mp"=>{"type"=>"country","tld"=>"mp","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Northern Mariana Islands","slds"=>[]}, "mq"=>{"type"=>"country","tld"=>"mq","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Martinique","slds"=>[]}, "mr"=>{"type"=>"country","tld"=>"mr","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Mauritania","slds"=>[]}, "ms"=>{"type"=>"country","tld"=>"ms","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Montserrat","slds"=>[]}, "mt"=>{"type"=>"country","tld"=>"mt","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Malta","slds"=>["com.mt","org.mt","net.mt","edu.mt","gov.mt"]}, "mu"=>{"type"=>"country","tld"=>"mu","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Mauritius","slds"=>[]}, "mv"=>{"type"=>"country","tld"=>"mv","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Maldives","slds"=>["aero.mv","biz.mv","com.mv","coop.mv","edu.mv","gov.mv","info.mv","int.mv","mil.mv","museum.mv","name.mv","net.mv","org.mv","pro.mv"]}, "mw"=>{"type"=>"country","tld"=>"mw","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Malawi","slds"=>[]}, "mx"=>{"type"=>"country","tld"=>"mx","2nd_level_registration"=>false,"foreign_registration"=>true,"country"=>"Mexico","slds"=>["com.mx","net.mx","org.mx","edu.mx","gob.mx"]}, "my"=>{"type"=>"country","tld"=>"my","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Malaysia","slds"=>["com.my","net.my","org.my","gov.my","edu.my","mil.my","name.my"]}, "mz"=>{"type"=>"country","tld"=>"mz","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Mozambique","slds"=>["co.mz","org.mz","gov.mz","edu.mz"]}, "na"=>{"type"=>"country","tld"=>"na","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Namibia","slds"=>[]}, "nc"=>{"type"=>"country","tld"=>"nc","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"New Caledonia","slds"=>[]}, "ne"=>{"type"=>"country","tld"=>"ne","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Niger","slds"=>[]}, "nf"=>{"type"=>"country","tld"=>"nf","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Norfolk Island","slds"=>[]}, "ng"=>{"type"=>"country","tld"=>"ng","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Nigeria","slds"=>["com.ng","org.ng","gov.ng","edu.ng","net.ng"]}, "ni"=>{"type"=>"country","tld"=>"ni","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Nicaragua","slds"=>["gob.ni","co.ni","com.ni","ac.ni","org.ni","nom.ni","net.ni","mil.ni"]}, "nl"=>{"type"=>"country","tld"=>"nl","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Netherlands","slds"=>[]}, "no"=>{"type"=>"country","tld"=>"no","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Norway","slds"=>[]}, "np"=>{"type"=>"country","tld"=>"np","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Nepal","slds"=>["com.np","org.np","edu.np","net.np","gov.np","mil.np"]}, "nr"=>{"type"=>"country","tld"=>"nr","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Nauru","slds"=>[]}, "nu"=>{"type"=>"country","tld"=>"nu","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Niue","slds"=>[]}, "nz"=>{"type"=>"country","tld"=>"nz","2nd_level_registration"=>false,"foreign_registration"=>true,"country"=>"New Zealand","slds"=>["ac.nz","co.nz","geek.nz","gen.nz","maori.nz","net.nz","org.nz","school.nz","cri.nz","govt.nz","iwi.nz","parliament.nz","mil.nz"]}, "om"=>{"type"=>"country","tld"=>"om","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Oman","slds"=>["com.om","co.om","edu.om","ac.om","sch.om","gov.om","net.om","org.om","mil.om","museum.om","biz.om","pro.om","med.om"]}, "pa"=>{"type"=>"country","tld"=>"pa","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Panama","slds"=>["net.pa","com.pa","ac.pa","sld.pa","gob.pa","edu.pa","org.pa","abo.pa","ing.pa","med.pa","nom.pa"]}, "pe"=>{"type"=>"country","tld"=>"pe","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Peru","slds"=>[]}, "pf"=>{"type"=>"country","tld"=>"pf","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"French Polynesia","slds"=>[]}, "pg"=>{"type"=>"country","tld"=>"pg","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Papua New Guinea","slds"=>["com.pg","net.pg","ac.pg","gov.pg","mil.pg","org.pg"]}, "ph"=>{"type"=>"country","tld"=>"ph","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Philippines","slds"=>[]}, "pk"=>{"type"=>"country","tld"=>"pk","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Pakistan","slds"=>[]}, "pl"=>{"type"=>"country","tld"=>"pl","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Poland","slds"=>[]}, "pm"=>{"type"=>"country","tld"=>"pm","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Saint Pierre and Miquelon","slds"=>[]}, "pn"=>{"type"=>"country","tld"=>"pn","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Pitcairn Islands","slds"=>[]}, "pr"=>{"type"=>"country","tld"=>"pr","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Puerto Rico","slds"=>[]}, "ps"=>{"type"=>"country","tld"=>"ps","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Palestine","slds"=>[]}, "pt"=>{"type"=>"country","tld"=>"pt","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Portugal","slds"=>[]}, "pw"=>{"type"=>"country","tld"=>"pw","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Palau","slds"=>[]}, "py"=>{"type"=>"country","tld"=>"py","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Paraguay","slds"=>["org.py","edu.py","mil.py","gov.py","net.py"]}, "qa"=>{"type"=>"country","tld"=>"qa","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Qatar","slds"=>[]}, "re"=>{"type"=>"country","tld"=>"re","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Réunion","slds"=>[]}, "ro"=>{"type"=>"country","tld"=>"ro","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Romania","slds"=>[]}, "rs"=>{"type"=>"country","tld"=>"rs","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Serbia","slds"=>[]}, "ru"=>{"type"=>"country","tld"=>"ru","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Russia","slds"=>[]}, "rw"=>{"type"=>"country","tld"=>"rw","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Rwanda","slds"=>[]}, "sa"=>{"type"=>"country","tld"=>"sa","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Saudi Arabia","slds"=>["com.sa","edu.sa","sch.sa","med.sa","gov.sa","net.sa","org.sa","pub.sa"]}, "sb"=>{"type"=>"country","tld"=>"sb","2nd_level_registration"=>false,"foreign_registration"=>true,"country"=>"Solomon Islands","slds"=>["com.sb","net.sb","edu.sb","org.sb","gov.sb"]}, "sc"=>{"type"=>"country","tld"=>"sc","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Seychelles","slds"=>[]}, "sd"=>{"type"=>"country","tld"=>"sd","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Sudan","slds"=>[]}, "se"=>{"type"=>"country","tld"=>"se","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Sweden","slds"=>[]}, "sg"=>{"type"=>"country","tld"=>"sg","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Singapore","slds"=>[]}, "sh"=>{"type"=>"country","tld"=>"sh","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Saint Helena","slds"=>[]}, "si"=>{"type"=>"country","tld"=>"si","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Slovenia","slds"=>[]}, # {"type"=>"country","tld"=>"sj","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Svalbard and Jan Mayen islands","slds"=>[]}, "sk"=>{"type"=>"country","tld"=>"sk","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Slovakia","slds"=>[]}, "sl"=>{"type"=>"country","tld"=>"sl","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Sierra Leone","slds"=>[]}, "sm"=>{"type"=>"country","tld"=>"sm","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"San Marino","slds"=>[]}, "sn"=>{"type"=>"country","tld"=>"sn","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Senegal","slds"=>[]}, # {"type"=>"country","tld"=>"so","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Somalia","slds"=>[]}, "sr"=>{"type"=>"country","tld"=>"sr","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Suriname","slds"=>[]}, "st"=>{"type"=>"country","tld"=>"st","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"São Tomé and Príncipe","slds"=>[]}, "su"=>{"type"=>"country","tld"=>"su","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Soviet Union","slds"=>[]}, "sv"=>{"type"=>"country","tld"=>"sv","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"El Salvador","slds"=>["edu.sv","gob.sv","com.sv","org.sv","red.sv"]}, "sy"=>{"type"=>"country","tld"=>"sy","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Syria","slds"=>[]}, "sz"=>{"type"=>"country","tld"=>"sz","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Swaziland","slds"=>[]}, "tc"=>{"type"=>"country","tld"=>"tc","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Turks and Caicos Islands","slds"=>[]}, "td"=>{"type"=>"country","tld"=>"td","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Chad","slds"=>[]}, "tf"=>{"type"=>"country","tld"=>"tf","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"French Southern Territories","slds"=>[]}, "tg"=>{"type"=>"country","tld"=>"tg","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Togo","slds"=>[]}, "th"=>{"type"=>"country","tld"=>"th","2nd_level_registration"=>false,"foreign_registration"=>true,"country"=>"Thailand","slds"=>["ac.th","co.th","in.th","go.th","mi.th","or.th","net.th"]}, "tj"=>{"type"=>"country","tld"=>"tj","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Tajikistan","slds"=>[]}, "tk"=>{"type"=>"country","tld"=>"tk","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Tokelau","slds"=>[]}, "tl"=>{"type"=>"country","tld"=>"tl","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"East Timor","slds"=>[]}, "tm"=>{"type"=>"country","tld"=>"tm","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Turkmenistan","slds"=>[]}, "tn"=>{"type"=>"country","tld"=>"tn","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Tunisia","slds"=>[]}, "to"=>{"type"=>"country","tld"=>"to","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Tonga","slds"=>[]}, "tp"=>{"type"=>"country","tld"=>"tp","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"East Timor","slds"=>[]}, "tr"=>{"type"=>"country","tld"=>"tr","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Turkey","slds"=>["nc.tr","com.tr","gen.tr","org.tr","biz.tr","info.tr","av.tr","dr.tr","pol.tr","bel.tr","mil.tr","bbs.tr","k12.tr","edu.tr","name.tr","net.tr","gov.tr","web.tr","tel.tr","tv.tr"]}, "tt"=>{"type"=>"country","tld"=>"tt","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Trinidad and Tobago","slds"=>[]}, "tv"=>{"type"=>"country","tld"=>"tv","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Tuvalu","slds"=>[]}, "tw"=>{"type"=>"country","tld"=>"tw","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Taiwan","slds"=>[]}, "tz"=>{"type"=>"country","tld"=>"tz","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Tanzania","slds"=>["co.tz","ac.tz","go.tz","or.tz","ne.tz"]}, "ua"=>{"type"=>"country","tld"=>"ua","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Ukraine","slds"=>["com.ua","gov.ua","net.ua","edu.ua","org.ua","in.ua"]}, "ug"=>{"type"=>"country","tld"=>"ug","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Uganda","slds"=>[]}, "uk"=>{"type"=>"country","tld"=>"uk","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"United Kingdom","slds"=>["ac.uk","co.uk","gov.uk","ltd.uk","me.uk","mod.uk","net.uk","nic.uk","nhs.uk","org.uk","plc.uk","police.uk","sch.uk","bl.uk","british-library.uk","icnet.uk","jet.uk","nls.uk","parliament.uk" ]}, "us"=>{"type"=>"country","tld"=>"us","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"United States","slds"=>[]}, "uy"=>{"type"=>"country","tld"=>"uy","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Uruguay","slds"=>["com.uy","edu.uy","gub.uy","net.uy","mil.uy","org.uy"]}, "uz"=>{"type"=>"country","tld"=>"uz","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Uzbekistan","slds"=>[]}, "va"=>{"type"=>"country","tld"=>"va","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Vatican City","slds"=>[]}, "vc"=>{"type"=>"country","tld"=>"vc","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Saint Vincent and the Grenadines","slds"=>[]}, "ve"=>{"type"=>"country","tld"=>"ve","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Venezuela","slds"=>["com.ve","edu.ve","gob.ve","mil.ve","net.ve","org.ve","info.ve","co.ve","web.ve","fuc.ve"]}, "vg"=>{"type"=>"country","tld"=>"vg","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"British Virgin Islands","slds"=>[]}, "vi"=>{"type"=>"country","tld"=>"vi","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"United States Virgin Islands","slds"=>[]}, "vn"=>{"type"=>"country","tld"=>"vn","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Vietnam","slds"=>[]}, "vu"=>{"type"=>"country","tld"=>"vu","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Vanuatu","slds"=>[]}, "wf"=>{"type"=>"country","tld"=>"wf","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Wallis and Futuna","slds"=>[]}, "ws"=>{"type"=>"country","tld"=>"ws","2nd_level_registration"=>true,"foreign_registration"=>true,"country"=>"Samoa","slds"=>[]}, "ye"=>{"type"=>"country","tld"=>"ye","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Yemen","slds"=>["com.ye","net.ye"]}, "yt"=>{"type"=>"country","tld"=>"yt","2nd_level_registration"=>true,"foreign_registration"=>false,"country"=>"Mayotte","slds"=>[]}, "za"=>{"type"=>"country","tld"=>"za","2nd_level_registration"=>false,"foreign_registration"=>true,"country"=>"South Africa","slds"=>["ac.za","city.za","co.za","edu.za","gov.za","law.za","mil.za","nom.za","org.za","school.za","alt.za","net.za","ngo.za","tm.za","web.za","bourse.za","agric.za","cybernet.za","grondar.za","iaccess.za","inca.za","nis.za","olivetti.za","pix.za","db.za","imt.za"]}, "zm"=>{"type"=>"country","tld"=>"zm","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Zambia","slds"=>["co.zm","org.zm","ac.zm"]}, "zw"=>{"type"=>"country","tld"=>"zw","2nd_level_registration"=>false,"foreign_registration"=>false,"country"=>"Zimbabwe","slds"=>["co.zw","ac.zw","org.zw"]} } def TLD.valid_tld?(t) # does the TLD exist? true or false tld(t).nil? ? false : true end def TLD.valid_sld?(d) # does the SLD exist? true or false sld(d).nil? ? false : true end def TLD.tld(d) # nil if invalid, otherwise string tld=d.split('.')[-1] t=@tld[tld] t.nil? ? nil : t["tld"] end def TLD.sld(d) # nil if invalid, otherwise string return nil if valid_tld?(d) == false a=tld_a(d) sld=d.split(".")[-2..-1].join(".") a["slds"].include?(sld) ? sld : nil end def TLD.tld_a(d) # returns an array of the tld @tld[d.split(".")[-1]] end def TLD.valid_domain?(d) extension(d).nil? ? false : true end def TLD.extension(d) # what domain is this in? tld + sld is needed. nil or string tld = tld(d) return nil if tld.nil? a=tld_a(d) if a["2nd_level_registration"] == true return tld else sld = sld(d) sld.nil? ? nil : sld end end def TLD.registered_name(d) # return just the registered name, like dogs from dogs.com or dogs.co.uk return nil if !valid_domain?(d) tld_len=extension(d).split(".").length d.split(".")[0..-(tld_len+1)][-1] end def TLD.domain_name(d) return nil if !valid_domain?(d) registered_name(d) + "." + extension(d) end def TLD.same_domain?(a,b) domain_name(a) == domain_name(b) end def tests # get the au array tlds=["au","nz","nn","edu"].each {|tld| domains=["yahoo","yahoo.com","yahoo.co"].each {|domain| d=domain+"."+tld puts "testing #{d}" if valid_domain?(d)==true puts "#{d} domain is valid" else puts "#{d} domain is invalid" end puts "domain ext is #{extension(d)}" puts "registered name is #{registered_name(d)}" # puts "tld is #{tld(d)}" =begin if valid_tld?(d)==true puts "#{d} tld is valid" else puts "#{d} tld is invalid" end if valid_sld?(d)==true puts "#{d} sld is valid" else puts "#{d} sld is invalid" end =end puts } } end end WhatWeb-0.4.9/lib/version_class.rb000066400000000000000000000045441320553762300170470ustar00rootroot00000000000000=begin Copyright 2009, 2017 Andrew Horton This file is part of WhatWeb. WhatWeb is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or at your option) any later version. WhatWeb is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with WhatWeb. If not, see . =end class Version def initialize(name_product = nil, versions = nil, url = nil) raise 'You must specify the name of the product' if name_product.nil? raise 'You must specify the available versions of the product' if versions.nil? raise 'You must specify the available url of the website' if url.nil? @name = name_product @versions = versions @files = Hash['filenames' => [], 'files' => [], 'md5' => []] @url = url @got_best_versions = false @best_versions = [] versions.each do |version| version[1].each do |file| next if @files['filenames'].include? file[0] @files['filenames'].push(file[0]) @files['files'].push(URI.join(@url.to_s, file[0]).to_s) status, url, ip, body, headers = open_target(@files['files'].last) @files['md5'].push(Digest::MD5.hexdigest(body)) end end end def best_matches return @best_versions if @got_best_versions == true @versions.each do |version| count = 0 version[1].each do |file| i = @files['filenames'].index(file[0]) count += 1 if @files['md5'][i] == file[1] end while !@best_versions.empty? && @best_versions[0][1] < count @best_versions.delete_at(0) end if count > 0 && (@best_versions.empty? || @best_versions[0][1] == count) and \ !@best_versions.include? [version[0], count] @best_versions.insert(0, [version[0], count]) end end @got_best_versions = true @best_versions.flatten! @best_versions.each_index { |i| @best_versions.delete_at(i + 1) }.sort! @best_versions end def matches_format self.best_matches if @got_best_versions == false @best_versions end end WhatWeb-0.4.9/my-plugins/000077500000000000000000000000001320553762300151775ustar00rootroot00000000000000WhatWeb-0.4.9/my-plugins/plugin-tutorial-1.rb000066400000000000000000000020141320553762300210160ustar00rootroot00000000000000## # This file is part of WhatWeb and may be subject to # redistribution and commercial restrictions. Please see the WhatWeb # web site for more information on licensing and terms of use. # http://www.morningstarsecurity.com/research/whatweb # Plugin.define "Plugin-Tutorial-1" do author "Your preferred name " description "Generic CMS is an open-source Content Management System developed in PHP." website "http://example.com/" # This is the matches array. # Each match is treated independently. # Matches # matches [ # This searches for a text string. { :text => "This page was generated by Generic CMS" }, # This searches for a regular expression. Note that the slashes are escaped. { :regexp => /This page was generated by Generic CMS<\/a>/ }, # This extracts the version of Generic CMS from the Mega generator tag. { :name => "Meta generator", :version => /" # 1999-12-31 version "0.1" description "GenericServer is an HTTP server for head-mounted devices that use the FOOT processor." website "http://example.com/" # This is the matches array. # Each match is treated independently. # Matches # matches [ # Title { :text => "Welcome to GenericServer" }, # HTTP Server Header # This detects the presence of the text "GenericServer" within the HTTP Server header { :regexp => /^GenericServer /, :search => "headers[server]" }, # HTTP Server Header # Version Detection # # This extracts the version of the "GenericServer" from the HTTP Server header # Note that many HTTP servers can be configured to hide the version so there are two patterns in this plugin { :version => /^GenericServer V([^\s]+)/, :search => "headers[server]" }, ] end WhatWeb-0.4.9/my-plugins/plugin-tutorial-3.rb000066400000000000000000000031461320553762300210270ustar00rootroot00000000000000## # This file is part of WhatWeb and may be subject to # redistribution and commercial restrictions. Please see the WhatWeb # web site for more information on licensing and terms of use. # http://www.morningstarsecurity.com/research/whatweb # ## # Version 0.2 # 1999-12-31 # Your Name # <----- updated from tutorial 1 # Added cookie detection ## Plugin.define "Plugin-Tutorial-3" do author "Your preferred name " # 1999-12-31 version "0.2" # <----- updated from tutorial 1 description "Generic CMS is an open-source Content Management System developed in PHP." website "http://example.com/" # The Google Dorks list is an array. This can help people find the product using a search engine. # Dorks # # <----- updated from tutorial 1 dorks [ '"Generic CMS login"', 'Generic login register linkname', ] # This is the matches array. # Each match is treated independently. # Matches # matches [ # This searches for a text string. { :text => "This page was generated by Generic CMS" }, # This searches for a regular expression. Note that the slashes are escaped. { :regexp => /This page was generated by Generic CMS<\/a>/ }, # This extracts the version of Generic CMS from the Mega generator tag. { :name => "Meta generator", :version => / "headers[set-cookie]", :regexp => /genericcms=[^;]+;/ }, ] end WhatWeb-0.4.9/my-plugins/plugin-tutorial-4.rb000066400000000000000000000012641320553762300210270ustar00rootroot00000000000000## # This file is part of WhatWeb and may be subject to # redistribution and commercial restrictions. Please see the WhatWeb # web site for more information on licensing and terms of use. # http://www.morningstarsecurity.com/research/whatweb # Plugin.define "Plugin-Tutorial-4" do author "Your preferred name " # 1999-12-31 version "0.1" description "GenericWAF is a commercial Web Application Firewall (WAF)." website "http://example.com/" # Matches # matches [ # HTTP Server Header { :search => "headers[server]", :regexp => /genericwaf\-nginx/ }, # Cookie { :search => "headers[set-cookie]", :regexp => /__genericwafuid/, :name=>"__genericwafuid cookie" }, ] end WhatWeb-0.4.9/my-plugins/plugin-tutorial-5.rb000066400000000000000000000031131320553762300210230ustar00rootroot00000000000000## # This file is part of WhatWeb and may be subject to # redistribution and commercial restrictions. Please see the WhatWeb # web site for more information on licensing and terms of use. # http://www.morningstarsecurity.com/research/whatweb # Plugin.define "Plugin-Tutorial-5" do author "Your preferred name " # 1999-12-31 version "0.1" description "GenericRouter is a home router using the ARM processor and busybox Linux." website "http://example.com/" # ShodanHQ results as at 1999-12-31 # # 1,234 for GenericRouter # Matches # matches [ # Title { :text => "GenericRouter" }, # The www-authenticate message { :search => "headers[www-authenticate]", :text => 'Basic realm="GenericRouter"' }, # Check MD5sum hashes of images to detect the exact version # In aggressive mode 3 these will be checked only if this plugin already matches # In aggressive mode 4 these will be checked anyway { :model => 'gsl2540b', :md5 => "d076eed06cafe1e4a74f83c7fdfe2e67", :url => '/generic/images/gsl2540b.jpg' }, { :model => 'gsl2640b', :md5 => "01aa666a65a72bb4ab0deadbeef525f4", :url => '/generic/images/gsl2640b.jpg' }, { :model => 'ggl3420', :md5 => "c3bb6c8124fe7106339cde087da6bb30", :url => '/generic/images/ggl3420.jpg' }, { :model => 'gwl2100ap', :md5 => "fa6350a0feedf00d9651c9aaf05f2187", :url => '/generic/images/gwl2100ap.jpg' }, { :model => 'gwl2230ap', :md5 => "71c307b6d7d82eeab5babe23c1ff41a9", :url => '/generic/images/gwl2230ap.jpg' }, { :model => 'gwl2700ap', :md5 => "3573c663c0ffeec53c0886518045a6f3", :url => '/generic/images/gwl2700ap.jpg' }, ] end WhatWeb-0.4.9/my-plugins/plugin-tutorial-6.rb000066400000000000000000000037511320553762300210340ustar00rootroot00000000000000## # This file is part of WhatWeb and may be subject to # redistribution and commercial restrictions. Please see the WhatWeb # web site for more information on licensing and terms of use. # http://www.morningstarsecurity.com/research/whatweb # Plugin.define "Plugin-Tutorial-6" do author "Your preferred name " # 1999-12-31 version "0.1" description "Describe what the plugin identifies" website "http://example.com/" # Dorks # dorks [ '"Generic CMS login"', 'Generic login register linkname', ] # Matches # matches [ # This searches for a text string. { :text => "This page was generated by Generic CMS" }, ] # You can write custom Ruby code in plugins for more control # There can be a passive function and an aggressive function. # The Passive function will always execute # ## # The following variables are available # # @body # @headers # @cookies # @status # @base_uri # @md5sum # @tagpattern # @ip ## def passive # make a matches array m=[] # If the HTTP status is 302 and the redirection location is /admin/genericcms.php then match if @status.to_s =~ /^302$/ and @headers["location"] =~ /^\/admin\/genericcms\.php$/ m << { :name => "302 redirection to /admin/genericcms.php" } end # You can add debugging and check the value of variables # pp @status # pp @headers # return the matches array, even if it's empty m end # Check other plugins with passive functions for examples. ## # The Aggressive function will only sometimes execute # At aggressive level 3 if a match is found, then the aggressive function executes # At aggressive level 4, the aggressive function always executes ## def aggressive # make a matches array. this returns the equivalent of the matches[] block above m=[] # return the matches array, even if it's emtpy m end ## Very few plugins need startup and shutdown functions # # This executes when the plugin is first loaded def startup end # This executes when the plugin is closed on whatweb shutdown def shutdown end end WhatWeb-0.4.9/my-plugins/plugin-tutorial-7.rb000066400000000000000000000136641320553762300210410ustar00rootroot00000000000000## # This file is part of WhatWeb and may be subject to # redistribution and commercial restrictions. Please see the WhatWeb # web site for more information on licensing and terms of use. # http://www.morningstarsecurity.com/research/whatweb ## # Version 0.3 # 2000-01-01 # Your name # Added plenty of example matches ## # Version 0.2 # 1999-12-31 # Your Name # <----- updated from tutorial 1 # Added cookie detection ## Plugin.define "Plugin-Tutorial-7" do author "Your preferred name " # 1999-12-31 version "0.3" description "Describe what the plugin identifies" website "http://example.com/" # Dorks # dorks [ '"Generic CMS login"', 'Generic login register linkname', ] # A comment block here is a good place to make notes for yourself and others # This is the matches array. # Each match is treated independently. ## # These are the symbols that can be used in matches. # ## Pattern Matching ## # :regexp -- A regular expression in Ruby format, eg. /^foobar$/ # :text -- Case insensitive text # :ghdb -- Google Hack Database format. This supports the use of intitle:, inurl: and minus. # :md5 -- The MD5 sum hash of the HTTP response body # :tagpattern -- A list of HTML tag names. # ## Where to Search ## # :search -- Can be "body" (default), "all", "headers", or "headers[x]" for a specific HTTP header # ## Naming the plugin match ## # :name -- You can optionally name the match. This name is displayed in verbose output. # ## Returning Data ## # Each of these symbols can be a regular expression or text. # # :version -- The version # :account -- A user account name # :module -- A module name # :make -- The make, e.g. NetGear # :model -- The model, e.g. SpeedErr # :firmware -- The firmware, e.g. 6.14.14 # :filepath -- A filepath. These can be displayed in error messages # :string -- This is to return data that isn't covered by the symbols above. E.g. an email address. # ## How certain is this pattern? ## # :certainty -- how certain is this match. 100 is certain (default), 75 is probably, and 25 is maybe # ## Limit the match to a URL path or an HTTP status ## # :url -- you can combine this with other variables or use by itself # :status -- The HTTP status of the response ## # Matches # matches [ # This searches for a text string. { :text => "This page was generated by Generic CMS" }, # This searches for a regular expression. Note that the slashes are escaped. { :regexp => /This page was generated by Generic CMS<\/a>/ }, # This searches for a text string and the match has a name that appears in verbose output { :name => "Meta generator", :text => ' 100 is certain' (default) # :certainty => 75 is probably # :certainty => 25 is maybe # :certainty is 25. If the title tag is matched, then maybe this webpage is the Generic CMS { :name => "title", :certainty => 25, :text => "Generic Links from Generic CMS" }, # :certainty is 75. This means that if the title tag is matched, then this webpage is probably the Generic CMS # This plugin match is split across multiple lines. This can aid readability. { :name => "title", :certainty => 75, :text => "Generic CMS Homepage" }, # check the presence of an HTTP header { :search => "headers[genericxxx]", :regexp => /^.*$/ }, # return the contents of an HTTP header { :search => "headers[genericversion]", :version => /^(.*)$/ }, # This returns the version. # The :version symbol is a regular expression. Whatever is found within the parenthesis is returned as the value of :version { :name => "Meta Generator", :version => / "Version in Meta Generator", :version => / 1. This returns the result of the second set of parentheses within the regular expression # The :module symbol is also a regular expression like :version. Note that the parenthesis is now after the text, "with modules" { :name => "Version in Meta Generator", :version => / 1 }, # :url can be used to limit a match to a URL path # This matches if both the URL path and the text is found # When WhatWeb is in aggressive mode, it will check if /admin/login.php exists { :url => '/admin/generic-cms-login.php', :text => 'Generic CMS Login Panel' }, # Generic CMS has a default favicon that displays the logo of the web application # In aggressive mdoe, the /favicon.ico path will be fetched # This matches if the HTTP response body, in this case the favicon.co image, has the follow MD5sum hash. { :url => "/favicon.ico", :md5 => '12dead87beef7f00d90cafed82babe5' }, # GHDB matches pages with a Google Hack Database format. # Very few plugins use the :ghdb symbol. { :ghdb => 'intitle:"Generic CMS" login register', :certainty => 25 }, # This match uses :tagpattern. This is a comma delimited list of all the HTML tags in the webpage. # Very few plugins use :tagpattern { :url => "/generic-login.php", :name => "Tag pattern for login page", :tagpattern => "!DOCTYPE,html,head,title,/title,meta,link,link,script,/script,meta,/head,body,div,h1,a,/a,/h1,form,p,label,br,input,/label,/p,p,label,br,input,/label,/p,p,label,input,/label,/p,p,input,input,input,/p,/form,p,a,/a,/p,p,a,/a,/p,/div,script,/script,/body,/html" }, ] end WhatWeb-0.4.9/plugin-development/000077500000000000000000000000001320553762300167115ustar00rootroot00000000000000WhatWeb-0.4.9/plugin-development/alexa-top-10.txt000066400000000000000000000001601320553762300215570ustar00rootroot00000000000000google.com youtube.com facebook.com baidu.com wikipedia.org yahoo.com google.co.in reddit.com qq.com taobao.com WhatWeb-0.4.9/plugin-development/alexa-top-100.txt000066400000000000000000000021761320553762300216500ustar00rootroot00000000000000google.com youtube.com facebook.com baidu.com wikipedia.org yahoo.com google.co.in reddit.com qq.com taobao.com tmall.com amazon.com twitter.com vk.com google.co.jp live.com instagram.com jd.com sohu.com sina.com.cn weibo.com 360.cn google.de google.co.uk google.com.br list.tmall.com google.fr google.ru yandex.ru linkedin.com netflix.com google.it google.es google.com.hk yahoo.co.jp google.com.mx google.ca pornhub.com alipay.com bing.com ebay.com twitch.tv t.co xvideos.com aliexpress.com imgur.com ok.ru wordpress.com microsoft.com office.com msn.com mail.ru hao123.com tumblr.com livejasmin.com imdb.com microsoftonline.com apple.com csdn.net blogspot.com detail.tmall.com stackoverflow.com google.com.au google.com.tr wikia.com deloton.com amazon.co.jp paypal.com google.com.tw github.com whatsapp.com google.co.id google.pl popads.net xhamster.com pinterest.com espn.com bongacams.com adobe.com diply.com coccoc.com google.com.ar mozilla.org amazon.de so.com googleusercontent.com dropbox.com txxx.com soso.com google.com.ua amazon.co.uk google.com.pk xnxx.com tianya.cn amazon.in nicovideo.jp savefrom.net google.com.eg pixnet.net bbc.co.uk WhatWeb-0.4.9/plugin-development/alexa-top-1000.txt000066400000000000000000000302671320553762300217320ustar00rootroot00000000000000google.com youtube.com facebook.com baidu.com wikipedia.org yahoo.com google.co.in reddit.com qq.com taobao.com tmall.com amazon.com twitter.com vk.com google.co.jp live.com instagram.com jd.com sohu.com sina.com.cn weibo.com 360.cn google.de google.co.uk google.com.br list.tmall.com google.fr google.ru yandex.ru linkedin.com netflix.com google.it google.es google.com.hk yahoo.co.jp google.com.mx google.ca pornhub.com alipay.com bing.com ebay.com twitch.tv t.co xvideos.com aliexpress.com imgur.com ok.ru wordpress.com microsoft.com office.com msn.com mail.ru hao123.com tumblr.com livejasmin.com imdb.com microsoftonline.com apple.com csdn.net blogspot.com detail.tmall.com stackoverflow.com google.com.au google.com.tr wikia.com deloton.com amazon.co.jp paypal.com google.com.tw github.com whatsapp.com google.co.id google.pl popads.net xhamster.com pinterest.com espn.com bongacams.com adobe.com diply.com coccoc.com google.com.ar mozilla.org amazon.de so.com googleusercontent.com dropbox.com txxx.com soso.com google.com.ua amazon.co.uk google.com.pk xnxx.com tianya.cn amazon.in nicovideo.jp savefrom.net google.com.eg pixnet.net bbc.co.uk google.com.sa google.co.th bbc.com craigslist.org thepiratebay.org cnn.com google.nl rakuten.co.jp ask.com nytimes.com soundcloud.com tribunnews.com youth.cn google.co.ve naver.com amazonaws.com onlinesbi.com google.co.za detik.com booking.com ebay.de salesforce.com google.co.kr stackexchange.com blogger.com thestartmagazine.com xinhuanet.com nih.gov vimeo.com quora.com theguardian.com spotify.com zhihu.com aparat.com slideshare.net google.gr fc2.com google.com.ph openload.co google.com.sg dailymotion.com chase.com chaturbate.com ebay.co.uk avito.ru google.se vice.com ettoday.net google.be gmw.cn roblox.com doubleclick.net fbcdn.net google.com.co cnet.com softonic.com mediafire.com google.co.ao hitcpm.com google.az iwanttodeliver.com google.cn globo.com dailymail.co.uk flipkart.com indeed.com alibaba.com cloudfront.net force.com walmart.com pinimg.com buzzfeed.com steamcommunity.com providr.com adf.ly tokopedia.com babytree.com google.at uol.com.br google.com.ng deviantart.com etsy.com godaddy.com twimg.com douyu.com w3schools.com blastingnews.com 1688.com discordapp.com wikihow.com washingtonpost.com 4chan.org google.com.vn amazon.it higheurest.com bet9ja.com mercadolivre.com.br china.com abs-cbn.com amazon.fr scribd.com redd.it google.ch people.com.cn google.com.pe google.cz messenger.com slack.com google.no china.com.cn google.pt blogspot.co.id google.ro google.cl metropcs.mobi redtube.com mama.cn hclips.com yelp.com gearbest.com sogou.com google.ae dkn.tv forbes.com ladbible.com bp.blogspot.com gfycat.com 39.net dingit.tv steampowered.com onlinevideoconverter.com bankofamerica.com cnblogs.com trello.com breitbart.com thesaurus.com ikea.com wikimedia.org youporn.com bestbuy.com thepennyhoarder.com bukalapak.com wetransfer.com google.ie files.wordpress.com wellsfargo.com youm7.com bilibili.com indiatimes.com codeonclick.com onoticioso.com myway.com mega.nz google.dz liputan6.com caijing.com.cn 9gag.com researchgate.net google.dk bitauto.com amazon.cn kakaku.com huanqiu.com foxnews.com speedtest.net rarbg.to livejournal.com google.hu coinmarketcap.com ebay-kleinanzeigen.de popcash.net battle.net huffingtonpost.com genius.com aliyun.com newstrend.news irctc.co.in hulu.com amazon.es google.fi leboncoin.fr eastday.com rambler.ru asos.com wordreference.com k618.cn blackboard.com zillow.com shutterstock.com kompas.com outbrain.com yts.ag spotscenered.info ltn.com.tw adexchangeperformance.com aol.com blogspot.in google.co.il allegro.pl gamepedia.com telegram.org sberbank.ru archive.org daum.net digikala.com tripadvisor.com businessinsider.com gostream.is kinopoisk.ru weebly.com pages.tmall.com torrentz2.eu subject.tmall.com 1337x.to mailchimp.com rutracker.org freepik.com buzzadnetwork.com amazon.ca ameblo.jp target.com spankbang.com livedoor.com medium.com goodreads.com ign.com zippyshare.com google.sk xfinity.com instructure.com zapmeta.ws gmx.net canva.com web.de nfl.com leagueoflegends.com olx.ua duckduckgo.com weather.com zendesk.com hola.com speakol.com varzesh3.com reverso.net hdzog.com momoshop.com.tw google.kz hicpm5.com skype.com ouo.io sourceforge.net ci123.com sabah.com.tr feedly.com flickr.com rednet.cn hotstar.com okdiario.com setn.com wordpress.org hp.com pipeschannels.com kaskus.co.id elpais.com fromdoctopdf.com ouedkniss.com userapi.com ntd.tv udemy.com orange.fr samsung.com kapanlagi.com livedoor.jp icloud.com streamable.com sciencedirect.com tfetimes.com kissanime.ru iqoption.com rt.com airbnb.com maka.im oracle.com haber7.com zipnoticias.com ndtv.com gamefaqs.com americanexpress.com ups.com rottentomatoes.com google.com.kw go.com usps.com taringa.net youdao.com eskimi.com capitalone.com beeg.com fiverr.com hdfcbank.com subscene.com alicdn.com uidai.gov.in onet.pl giphy.com dictionary.com humblebundle.com homedepot.com seasonvar.ru cricbuzz.com mmoframes.com taleo.net iqiyi.com nba.com pixabay.com siteadvisor.com glassdoor.com merdeka.com dmm.co.jp albawabhnews.com telegraph.co.uk ensonhaber.com mercadolibre.com.ar office365.com line.me yadi.sk taboola.com theverge.com rumble.com google.co.nz zoho.com intuit.com application-77my.com state.gov uptodown.com slickdeals.net doublepimp.com vidio.com wix.com atlassian.net accuweather.com doublepimpssl.com banvenez.com prezi.com free.fr repubblica.it wp.pl ultimate-guitar.com bloomberg.com themeforest.net independent.co.uk naver.jp banggood.com convert2mp3.net box.com wittyfeed.com t-online.de google.lk springer.com wish.com pandora.com mit.edu primosearch.com easypdfcombine.com behance.net patreon.com fedex.com usatoday.com bittrex.com adexchangecloud.com dell.com h8vzwpv.com rediff.com gismeteo.ru seznam.cz gyazo.com ebay.it suning.com kooora.com wiktionary.org google.bg vporn.com icicibank.com videoyoum7.com jianshu.com upwork.com goo.ne.jp sabq.org evernote.com wattpad.com mi.com cambridge.org blpmovies.com gamer.com.tw ytimg.com ebay.com.au shopify.com nextlnk1.com media.tumblr.com playstation.com myanimelist.net addthis.com espncricinfo.com brilio.net mercadolibre.com.mx ibm.com cnzz.com manoramaonline.com patch.com marca.com coinbase.com rutube.ru sex.com 17ok.com tutorialspoint.com lazada.co.id fwbntw.com perfecttoolmedia.com chegg.com hubspot.com buyma.com filehippo.com olx.pl quizlet.com elmundo.es okta.com onedio.com flvto.biz epochtimes.com libero.it bldaily.com webmd.com sahibinden.com wsj.com hm.com blogspot.mx google.com.do theepochtimes.com gizmodo.com spiegel.de rarbg.is wiley.com souq.com citi.com pikabu.ru google.by nike.com urbandictionary.com shaparak.ir ptt.cc thefreedictionary.com tistory.com indoxxi.net billdesk.com naukri.com drom.ru telewebion.com givemesport.com biobiochile.cl wixsite.com baike.com jrj.com.cn reimageplus.com thewhizmarketing.com blog.jp kickstarter.com piet2eix3l.com chatwork.com khanacademy.org friv.com pussl18.com digitalprivacyalert.org lie2anyone.com blueseek.com academia.edu list-manage.com discogs.com investopedia.com yandex.ua asana.com nownews.com hurriyet.com.tr velocecdn.com weblio.jp newegg.com 51sole.com reuters.com livescore.com 163.com google.com.ly emol.com att.com cerpen.co.id oload.stream chip.de gsmarena.com surveymonkey.com cnbc.com google.rs groupon.com macys.com google.com.mm ria.ru namnak.com thebalance.com adhoc2.net login.tmall.com asus.com cbssports.com trackingclick.net nur.kz xda-developers.com informationvine.com paytm.com 4shared.com comcast.net bandcamp.com drive2.ru nextlnk2.com nordstrom.com verizonwireless.com tube8.com videodownloadconverter.com linkshrink.net hespress.com mercadolibre.com.ve ck101.com zoom.us beytoote.com voot.com pantip.com bleacherreport.com ebc.net.tw moneycontrol.com adp.com squarespace.com chinaz.com baskino.co japanpost.jp goo.gl npr.org y8.com blibli.com eyny.com wowhead.com kinogo.cc google.hr almasryalyoum.com java.com fbsbx.com newtabtv.com umblr.com gmarket.co.kr xe.com bild.de stockstar.com inquirer.net gamespot.com youboy.com engadget.com ieee.org digitaldsp.com blogfa.com eventbrite.com mathrubhumi.com lifehacker.com wp.com lenta.ru flirt4free.com google.com.ec techcrunch.com stanford.edu uploaded.net hatena.ne.jp chinadaily.com.cn zhanqi.tv voc.com.cn lemonde.fr primevideo.com pchome.com.tw op.gg merriam-webster.com elmogaz.com google.tn as.com utorrent.com webex.com nametests.com nypost.com jeuxvideo.com grammarly.com infusionsoft.com mes5apps.com kijiji.ca yesky.com putrr18.com camdolls.com rapidgator.net junbi-tracker.com exoclick.com chaoshi.tmall.com superappbox.com getpocket.com milliyet.com.tr corriere.it 104.com.tw youjizz.com crunchyroll.com coursera.org t.me drudgereport.com ecosia.org focuusing.com disqus.com ea.com investing.com bitly.com olx.com.br azlyrics.com kotaku.com 2ch.net tvbs.com.tw allrecipes.com norton.com cdiscount.com goal.com yaplakal.com yourporn.sexy appledaily.com eksisozluk.com reallifecam.com timeanddate.com mellowads.com gosuslugi.ru animeflv.net lazada.com.my friendlyappz.com ask.fm grid.id fidelity.com hotels.com divar.ir latimes.com lapatilla.com discover.com sarkariresult.com techradar.com mawdoo3.com gongchang.com wunderground.com expedia.com thesun.co.uk google.com.my rbc.ru fanserials.tv intoday.in autodesk.com lefigaro.fr gstatic.com costco.com idntimes.com time.com europa.eu healthline.com lenovo.com gogoanime.io ccm.net agreemand.com 1111.tmall.com banesconline.com jimdo.com badoo.com uzone.id express.co.uk meetup.com chron.com caliente.mx tempo.co tabelog.com ssl-images-amazon.com secureserver.net uptobox.com kinokrad.co ebay.fr mobile01.com thewhizproducts.com cqnews.net jw.org pinterest.co.uk lifewire.com redirectvoluum.com segmentfault.com hawaaworld.com yalla-shoot.com interia.pl coursehero.com subito.it discuss.com.hk 4pda.ru thehill.com google.lt voyeurhit.com qingdaonews.com udn.com ebay.in ca.gov hatenablog.com google.iq harvard.edu nature.com m62-genreal.com 5ch.net ajel.sa wtoip.com google.tm ruten.com.tw issuu.com elbalad.news bhphotovideo.com kumparan.com lazada.com.ph kizlarsoruyor.com intel.com pussl32.com vnexpress.net motherless.com cisco.com gov.uk mangastream.com comicbook.com mobafire.com southwest.com bestadbid.com kohls.com torrents9.pe smallpdf.com vidzi.tv boredpanda.com kissasian.ch 58.com usnews.com abourselfi.com playmediacenter.com fatosdesconhecidos.com.br zara.com lazada.co.th psu.edu okcupid.com alodokter.com premierleague.com ticketmaster.com exosrv.com el-nacional.com allocine.fr liveinternet.ru zone-telechargement.ws myfreecams.com commentcamarche.net bancodevenezuela.com polygon.com dmm.com avast.com bmovies.to dream.co.id realtor.com v3rjvtt.com neobux.com deviantart.net nocookie.net mobile.de google.com.gt doorblog.jp politico.com vubird.com gap.com eanswers.com bankmellat.ir nbcnews.com uniqlo.tmall.com xtube.com fanfiction.net calcch.com toutiao.com metropoles.com 11st.co.kr artstation.com bitbucket.org perfectgirls.net okezone.com nikkei.com theatlantic.com uniqlo.com yandex.kz google.com.af google.si ly.com cam4.com hootsuite.com piriform.com nhl.com sportbible.com airtel.in olx.in freejobalert.com rvcj.com preligions.com sinoptik.ua python.org instructables.com epwk.com suara.com douban.com hamariweb.com prnt.sc tomshardware.com duolingo.com n11.com lowes.com popmyads.com sputniknews.com mirror.co.uk yespornplease.com mercari.com donga.com mashable.com namu.wiki gazetaexpress.com food.tmall.com internetspeedtracker.com jiameng.com tomsguide.com superuser.com zougla.gr zf.fm mathworks.com istockphoto.com best2017games.com ukr.net elsevier.com labanquepostale.fr xbox.com bodybuilding.com chess.com sapo.pt otvfoco.com.br ninisite.com eztv.ag bitmedianetwork.com dafont.com asahi.com kayak.com td.com pexels.com biblegateway.com clipconverter.cc myanmarload.com sephora.com qualtrics.com zing.vn gmanetwork.com makemytrip.com bamilo.com tradingview.com cbsnews.com abcnews.go.com cinecalidad.to hepsiburada.com telegraf.com.ua cnnindonesia.com avg.com forgeofempires.com kompasiana.com turbobit.net infourok.ru tnt-online.ru bookmyshow.com ecollege.com justdial.com internetdownloadmanager.com news.com.au rozetka.com.ua android.com lavanguardia.com study.com jumia.com.ng blockchain.info quizzstar.com americanas.com.br aptitudemedia.co scribol.com cbc.ca yandex.com.tr agoda.com nhk.or.jp zol.com.cn liveadexchanger.com indiamart.com gazzetta.it gotporn.com caixa.gov.br WhatWeb-0.4.9/plugin-development/charset-test-list.txt000066400000000000000000000001571320553762300230340ustar00rootroot00000000000000www.amazon.co.jp www.pravda.ru www.118114.cn 360.cn www.cntv.cn fastpic.ru http://www.columbia.edu/~fdc/utf8/ WhatWeb-0.4.9/plugin-development/find-common-stuff000077500000000000000000000043711320553762300221770ustar00rootroot00000000000000#!/usr/bin/env ruby # helper for plugin research # identify common tags, paths / quoted text in tags, link text # add paths, truncated paths # require 'getoptlong' require 'pp' ignore_tags=%w| ' }, { :text=>'Article Publisher PRO Administrator Control Panel' }, { :text=>'
Admin Control Panel
' }, { :text=>'
' }, # Error page { :text=>'

Please use a proper method to browse article(s) - The method you are using is not allowed...
' }, # Version detection # Powered by text { :version=>/
Powered by Article Publisher PRO<\/a> v([\d\.]+)/ }, ] end WhatWeb-0.4.9/plugins/artiphp-cms.rb000066400000000000000000000021501320553762300173260ustar00rootroot00000000000000## # This file is part of WhatWeb and may be subject to # redistribution and commercial restrictions. Please see the WhatWeb # web site for more information on licensing and terms of use. # http://www.morningstarsecurity.com/research/whatweb ## Plugin.define "Artiphp-CMS" do author "Brendan Coles " # 2012-05-17 version "0.1" description "Artiphp CMS - open source CMS - Requires PHP" website "http://www.artiphp.com/" # Google results as at 2012-05-17 # # 251 for "Artiphp" "2001" "est un logiciel libre" "sous licence GPL" # Dorks # dorks [ '"Artiphp" "2001" "est un logiciel libre" "sous licence GPL"' ] # Matches # matches [ # HTML Comments { :text=>'' }, { :text=>'' }, { :text=>'' }, # Version Detection # Copyright Footer { :version=>/